Results 1 to 9 of 9
  1. #1
    Star Lounger
    Join Date
    Mar 2018
    Location
    Australia
    Posts
    50
    Thanks
    23
    Thanked 0 Times in 0 Posts

    Beautifulsoup download for win 7

    I am trying to learn Python, I am at absolute, very absolute beginner, probably just past the "hello world" bit.

    I don't care about the 'hello world" phase of the game, I just want to scrape a web page the same way VBA does, only in Python though.

    So, having downloaded ans installed the latest version of Python, done some examples,( woopy do) , now I am ready to to do a web query, to replicate what my current Excel BA does and has been doing so without a glitch for the past 10 years +

    Question is, where the heck do I download beutifulsoup, it is a required set of librarys, for Python, so it can do web scrape, to use their termologies.

    I have tried google, but it seems the downloads and installs are complicated, they don't seem to have installers etc. It's all mush, makes no sense.

    Anyone know ?

    Thanks

  2. #2
    WS Lounge VIP
    Join Date
    Dec 2009
    Location
    Earth
    Posts
    9,055
    Thanks
    65
    Thanked 1,144 Times in 1,065 Posts
    This site has a nice description.
    http://www.scrapingauthority.com/201...-beautifulsoup

    cheers, Paul

  3. The Following User Says Thank You to Paul T For This Useful Post:

    XP_7 (2018-07-13)

  4. #3
    Star Lounger
    Join Date
    Mar 2018
    Location
    Australia
    Posts
    50
    Thanks
    23
    Thanked 0 Times in 0 Posts
    Thanks, will try

  5. #4
    3 Star Lounger
    Join Date
    Dec 2009
    Location
    Surrey, UK
    Posts
    268
    Thanks
    12
    Thanked 64 Times in 58 Posts
    Quote Originally Posted by XP_7 View Post
    I am trying to learn Python, I am at absolute, very absolute beginner, probably just past the "hello world" bit. Question is, where the heck do I download beutifulsoup, it is a required set of librarys, for Python, so it can do web scrape, to use their termologies.
    I had to re-acquaint myself with the python interface, and, as you found, there are lots of questions and answers out there, but which ones apply to running Python under windows. In the process, I read about PIP (I think installed with python, or it may have come with the IDLE IDE). So how to use PIP? The answer seems to be from the Windows command line:

    C:\Program Files\Python>pip list
    Package Version
    ---------- -------
    pip 10.0.1
    setuptools 18.2

    Note that originally I had PIP v7.1.2 and every time I typed a PIP command, it nagged me to update, so I did:
    C:\Program Files\Python>python -m pip install --upgrade pip
    (which I pasted from the nag text), and v10 was installed. How to install Beautifulsoup? Simples:
    C:\Program Files\Python>pip install beautifulsoup4

    and it goes off to the internet and downloads and installs it! To check:

    C:\Program Files\Python>pip list
    Package Version
    -------------- -------
    beautifulsoup4 4.6.0
    pip 10.0.1
    setuptools 18.2

    Now if that info was on the web, I didn't come across it before just having a go. Note that I have no idea what to do next: how to use BS functions etc. Good luck, hth, Martin

    PS That is all covered (and more) in Paul's link - but finding out kept me busy.
    Last edited by mngerhold; 2018-07-07 at 05:31.

  6. The Following 2 Users Say Thank You to mngerhold For This Useful Post:

    Lugh (2018-07-28),XP_7 (2018-07-13)

  7. #5
    Star Lounger
    Join Date
    Mar 2018
    Location
    Australia
    Posts
    50
    Thanks
    23
    Thanked 0 Times in 0 Posts
    Thanks, will try soon.

    have asked in other places and general reply is, "welcome to the world of programming".
    yeah riiight, MS has spoilt the majority it seems with download/click/install and voila !

  8. #6
    Star Lounger
    Join Date
    Mar 2018
    Location
    Australia
    Posts
    50
    Thanks
    23
    Thanked 0 Times in 0 Posts
    I'm dropping out trying to learn Python.

    I'll try to illustrate, abstractly...

    What I do and have accomplished here, ( credits to many members ) I use Excel VBA.
    It is not "perfect", but does the job.
    By "not perfect", I mean it's mostly learnt from Record Macro and then clean up or trim the recording in the VBA editor.
    It's a "rat -rod" of a process, but gets to the finish line, and Loops.
    The "proffesional" version was once built by one who is a programmer, he retired, therefore I was left with nothing, absolutely nothing that would work.
    All I was left with was my script. That script required the programmer's program to Run. My calculations though came from Excel.
    Too long a story to explain.
    Luckily someone showed me how to Record Macro external data, and tweaked that recording, the rest happened over time.
    Soon, that web query, or "get external data" from a web page may not work with VBA. It's a JSON issue.

    I thought, Python community will help me solve this providential glitch.

    But no, I'm expected to start with the "hello worlds", and make some useless games, and use letters of the Greek alphabet, all in the name of "correctness", or just because it has to be "priffesional corporate industry standard and scientific level"
    yeah that right, become a computer scientist, read volumes of methods, totally unrelated to what I want to achieve, just do I can "fish", rather than "handed a fish".... well, I don't fish, I make a better fishing rod, or, use a net, you will catch more fish !
    See ?

    I don't need to catch the ocean of fish in the world of Python code jjst to do a web query conversion because someone decided to go from HTML to whatever, JSON I don't really care....

    My Excel and VBA's now does a bettr job than a Firebird server with relational database and script editor. Cos that's the programmer used before he retired, and left me hanging with a dormant script in a BASIC language unique to that program ! ( and the money I paid ).

    When the day arrives and the web queries or "Get External Data from HTML no longer communicate with VBA, then I'll cross bridge.

    There is no way I can do a full blown computer science course with some Greek ( lamda keystrokes ) alphabet, online so the "hello worlds" have all the nice trimmings with correctness.

    I have a "rat rod" of a Excel VBA method, and it works for me.
    Again, I that this forum. could not have achieved the end result to do my real-time sports probability analysis.

    Thank you.

    Some rat-rod pictures, ( My excel work book/s are distant cuzins )

    http://ratrodsrule.tumblr.com/

  9. #7
    3 Star Lounger
    Join Date
    Dec 2009
    Location
    Surrey, UK
    Posts
    268
    Thanks
    12
    Thanked 64 Times in 58 Posts
    Some questions:
    1). Did the 'professional' leave you with the source code for his program (PP) - he should have, given that you apparently paid for it (unless he specifically excluded that in whatever contract you had).
    2) Why were you left with 'nothing' when he retired? Surely the PP continued to work?
    3) It seems your script in VBA interacts with the PP, so what makes you think the PP won't continue to work?
    4) Do you have any reason to expect 'Get external data' to stop cooperating with VBA?

    I agree with your sentiment that you just want to get something working in Python - but I did find it quite different to what I was familiar with (in some respects, but similar in others). I would still like to know exactly what the professional left you with (what databse uses BASIC scripts, apart from MS Access?), but realise it's none of my business.

    atb, Martin

  10. #8
    Star Lounger
    Join Date
    Mar 2018
    Location
    Australia
    Posts
    50
    Thanks
    23
    Thanked 0 Times in 0 Posts
    mngerhold
    Some questions:
    1). Did the 'professional' leave you with the source code for his program (PP) - he should have, given that you apparently paid for it (unless he specifically excluded that in whatever contract you had).
    2) Why were you left with 'nothing' when he retired? Surely the PP continued to work?
    3) It seems your script in VBA interacts with the PP, so what makes you think the PP won't continue to work?
    4) Do you have any reason to expect 'Get external data' to stop cooperating with VBA?
    1. The program component by component was made up of
    a: Firebird Server with relational database and script editor.
    http://firebirdsql.org/
    My part of the deal is in the script. I invited the programmer over, showed my idea and some program calculators and basically wanted it all
    automated. ( At the time XP just came onto the scene ).
    My first quote from a IT company was at the time $75,000.00 plus ongoings. So I passed on my idea ever seeing the light of day.
    By a stroke of luck one night at the bar, this dude asked me I was so glump, told him and said the works for a small IT company.
    That quote was $5,000.00 and a new computer thrown in.
    Reason ? We were on the same page so to speak, other clients wanted similar programs, so why not just have one server_database, with a
    script editor. Clients learn to write their own scripts, while the small IT firm simply work on their end.
    I would download the entire program from his main server, he wrote my first script, basically. And I was suppose to learn it.
    What was to take 3 months, took 2 years, no extra charge. because my script was complex and initially the server we downloaded had to be
    redesigned, due to the demand of the script, and probably the scripts of other clients. ( I think the company went broke due to all this and the
    workers left, the rented unit became vacant, and I had to hunt him down and he finished my program/script, then probably had a nervous
    breakdown ) We sort of have a mutual respect for each other, and that is when I started to understand what IT'ing is all about, from the outside
    looking in.

    2. a:Left with nothing due to the source of the data becoming on-line via Internet. The raw data used to come via TV arial and decoder box or
    Decoder card. This was inevitable, subscription for raw data feeds was once sourced, and they have the national monopoly, they fed it to the TV
    stations betting agencies at various public outlets. Now it's all INTERNET, no more TV aerial cables tapping freely into the signal. Hence "digital
    economy".
    The signal was the same signal where you have sub-titles for hearing impaired. Those days there was lots of sub-channels and that's where
    the raw data came from, the sports market section, prices updating all the time. Including stock markets.
    When that service ended nation wide, zilch---no data feed to the firebird server via TV ariel cable_decoder box and a cable into the PC
    unless the programmer updated it, which never happened.
    Therefore I'm, and others, are/were left with a few scripts, but no program to play them because no data feed_IN or server_database_script-
    editor to do my thing.
    b: It continued to work till one day the HD crashed, I phoned to get it re-installed, but no longer in the business.

    3. My script, actually starts with "Global_Variables".
    Then it has settings and all sorts of things to do BEFORE the event, and AFTER the event, ( Sell-Sold_Profit or Loss )
    Hedge calculations and strategies and a timer/s
    The Timer means, if the event starts at a fixed time of 11.50AM and it's 11.48 AM, then discard the event/
    Therefore in my settings I would put -120 seconds.



    In my VBA project, I found a way to subtract time, and with series of If Then VBA coding, I obtain the same calculation.
    Therefore the Events List listed in time order from first to last event of the day is text format when imported from the web.
    I would then split the computer's time and re-assemble the hour-minutes in text format, then calculate that into seconds.
    So a subtractions of 11.50 - 11.48 is 2, * 60 is 120 seconds.( If Cell A1 =119, 0 ) Then do this or that, unless it's 1, do something else.

    4) Do you have any reason to expect 'Get external data' to stop cooperating with VBA?
    YES,
    because the data source from the web is at this particular web site is progressivley going from HTML to JSON or some new glossy feature.
    VBA does not seem to do JSON web query, I have tried.

    part of my Excel VBA project use Python to log into the web site and do transactions.
    Well, not directly VBA to Python, it's: VBA_LOG-A-TEXT-FILE_BAT_PYTHON_READ-THE TEXT-FILE_SEND-TO-JSON.

    I am not a programmer.
    Last edited by XP_7; 2018-07-28 at 06:00.

  11. #9
    Star Lounger
    Join Date
    Mar 2018
    Location
    Australia
    Posts
    50
    Thanks
    23
    Thanked 0 Times in 0 Posts
    mngerhold
    Reply to ( regarding script language )

    3) It seems your script in VBA interacts with the PP, so what makes you think the PP won't continue to work?
    This is a snippet of the script/code. It's a section where user controls a manual function, clearing

    Code:
    sub ClearTheEvent
       '*******************************************************************
       ' this function designed to clear Store table of temp info
       ' since race data no longer needed - resetting
       '*******************************************************************
       dim i
       dim Store
       dim number_selected
       Store = TVarStore.Create
       Store.Reference = Reference
       try
          if Store.ReadValue("NumberSelected",number_selected) >= 0 then
             for i = 1 to number_selected
                Store.DeleteValue("Selection" + IntToStr(i))
                Store.DeleteValue("Bet" + IntToStr(i))
                Store.DeleteValue("Expect" + IntToStr(i))
             next
          end if
          Store.DeleteValue("EventCode")
          Store.DeleteValue("WaitTime")
          Store.DeleteValue("NumberSelected")
          Store.DeleteValue("Outlay")
       finally
          Store.Free
       end
    end sub
    At the time, we the clients/ customers, all we had to do is learn this simple language using the script editor.

    Back then I knew absolutely nothing about writing this type of language, but he did say, "you *can* learn this".
    I nodded and just said, "hmm" lol
    Now I know better, I could have learnt it

    Store refers to the database, the data it is using for the current event. So it is clearing that event in the time order so it can get onto the next event
    in the database.

    WaitTime means the time allowed to wait after the event, sometimes due to unforseen circumstances beyond the control of the users,
    there has to be a max time wait for results. Results carry numerical factors, these factors are used for calculations in regards to bookeping, if something happens at the event venue, it mat take hours to have a result, therefore have that max times cleared, so the program can continue.

    All this logical process is my part, what I want my script to do.
    Once the script language is learnt and practiced, then write as many scripts as required..endless.
    And the programmer/s just update and service their end of the deal.

    In reality, for this type of sport in question, one person alone cannot do this with limited funding and resources.
    I think I was the only customer left because this was a huge task, trial and error wise.

    But I knew it can be done, hence Excel VBA and my project became "server_database-web-qeurier_etc etc etc etc _all in one.
    A rat-rod prototype.

    I actually got back in touch with the programmer to invite him over for coffee and see my creation.
    Declined.
    :-)

    The only thing left for me to do, is to print to dot matrix, and sounds.

    I had sounds at certain parts of the script, such as " ding ding" if a event suited the parameters, or passed the filters
    sound file for other things "important". reason for this was as the process was working in the background, I did not have to be glued to the screen.
    A sound would happen and depending what sound.wav, I'd get up and check things.

    dot matrix printer are obsolete these days, but there was a box full of endless stream A4 paper with those holes on either end fed into the printer.
    Have not got around to coding that yet, no point really cos none of those printers around anymore.

    And finally, I wanted it to send me data via email or mobile phone message, so I know whats happening while away, or at the actual event.
    I do have VBA code to send to email, but not implemented at this point.

    So yeah, Python may solve most of this, but ain't gonna happen, there really is nothing more to prove, VBA got me over the line, got my stats. I wanted to analyze and finally made a personal conclusion.

    In probability, work on averages and find the percentage within one's own comfort zone towards a + positive result and then stop, because for some reason, the - negative will take it all back.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •