Skip Navigation

InitialsDiceBearhttps://github.com/dicebear/dicebearhttps://creativecommons.org/publicdomain/zero/1.0/„Initials” (https://github.com/dicebear/dicebear) by „DiceBear”, licensed under „CC0 1.0” (https://creativecommons.org/publicdomain/zero/1.0/)S

solrize

@ solrize @lemmy.ml

Posts
19
Comments
879
Joined
3 yr. ago

  • Having trouble downloading a forum thread with wget and httrack

    Jump

  • FFF won't work on that site right out of the box, because of the bot challenge. You'll need a workaround. FFF has the same problem with fanfiction.net (FFN) which is one of the biggest fanfic sites. So there are a number of FFF github issues and doc entries related to FFN and looking at those might help. I do know that FFN is scrapable using browser orchestration. https://github.com/FicHub/fichub.net may have some code for that, but getting your own instance running will be quite a bit more headache than just running fanficfare.

  • Having trouble downloading a forum thread with wget and httrack

    Jump

  • I tried wget and got a small index file that looks like a bot challenge, maybe Cloudflare Turnstile. I don't know of a simple automated workaround. But, Turnstile gives you a rewritten url that then goes back to the forum page and sets a cookie, iirc. So if you can visit the page with a real browser, then save the cookie and transfer it to wget (there's some option to set an arbitrary header) that's one thing to try.

    There's another hack used by fanfiction.net readers, where if you want to save a multi-page story, you can manually visit each page with a browser, i.e. click "next" again and again to load all the pages. Up to a few dozen such clicks isn't so bad. Fanficfare (fanfic downloading program, sometimes abbreviated FFF, https://github.com/JimmXinu/FanFicFare ) then has an option to retrieve the pages from your on-disk browser cache instead of trying to get them from the remote server. That's another approach you can try, either with fanficfare or your own scripts.

    The site you're looking at uses xenforo which is a very popular forum server program. The actual layout of xenforo sites varies, but fanficfare probably already recognizes something similar, so try using one of those interfaces. I think spacebattles.net (another fic site) uses xenforo and FFF supports it, so it might be a good start. You will have to modify FFF to recognize civfanatics instead of spacebattles. It will help to know or pick up some Python, but you shouldn't have to become an expert.

    Added: if you really want to automate your scrape, you will have to orchestrate a browser as mentioned earlier. IDK if there is code around to already do it. If you can program, it's not terribly hard to use Puppeteer or Selenium, but it will take some farting around to deal with the site layout and anti-bot stuff. It's not guaranteed to work right off the bat, but with enough determination you can do it, especially if your scrape volume is low and you can run it slowly. I'm not deeply involved in this stuff (like you, I just occasionally want to download something for personal use) but there are tons of webpages and articles by people (who I'd mostly consider evil) who do it at scale.

  • Having trouble downloading a forum thread with wget and httrack

    Jump

  • You're trying to run a scraper, and unfortunately a lot of AI companies are doing the same thing on such a big scale that it creates a DOS attack. So lots of sites now have anti-scraping measures. I clicked the forum link and saw a brief interstitial that looked like bot protection.

    I'm unfamiliar with httrack but wget fails pretty often by the site just rejecting the user agent. Try "wget -Dfoo [url]" to save the response headers in the file "foo", if I remember it right. That will let you check if there is an error code. You could also examine the too-small html index that you got, to see if it has error messages inside.

    Sometimes curl works when wget fails. For both of them, there are CLI options to set the user agent to something different.

    Getting images from forums often requires you to have a login cookie in your client. It's simplest to log into the site and then paste the cookie into your scraping program or script.

    The next thing after wget/curl would be to write a scraping script (say with python urrlib) that can analyze the html a little as it goes. The thing after that would be script an actual browser, with puppeteer or selenium etc.

    Regarding the links pointing to your local disk, that's probably because they are relative links, like "href=./foobar.html". You can make them point to the remote server by inserting an HTML BASE tag into the top of the file, like

    <BASE HREF="https://whateveritwas.com/forum">

    or whatever. If it's just for one or two pages you can do that manually. Otherwise, modify your scraping script to insert it, before or after saving the output.

  • What's a fun piece of free software I should learn to use this week?

    Jump

  • Emacs! Emacs Makes All Computing Simple.

  • What are your favorite joke/insult names for corporations?

    Jump

  • Some friendlier ones. There was once an expensive camera brand called "Exakta" but Exakta owners affectionately called them "Approxima".

    I worked near a slightly fancy Chinese restaurant called Royal East but my coworkers (who went there often) liked to call it "Royal Eats".

    There was a nearby supermarket called Purity Supreme but people called it Poverty Supreme. That was waaaay before Whole Foods.

    Amherst, Massachusetts was sometimes anagrammed to "Hamster". I mean you could address letters that way and they'd get there.

  • Exploring the "Dario and Amanda" Prompt | Alec Armbruster - 'Claude Opus 5 may have just leaked a trove of private 2024–2025 chat logs from Anthropic employees'

    Jump

  • Who and who?

  • RFK Jr launches new cooking show with sauerkraut-inspired recipes

    Jump

  • Kash Patel Loses Lawsuit Against Man Who Called Him “Googly-Eyed” Chud

    Jump

  • Well I use duckduckgo instead of google. Maybe that makes me a ducky-eyed chud, heh.

  • Kash Patel Loses Lawsuit Against Man Who Called Him “Googly-Eyed” Chud

    Jump

  • We have a whole community "degoogle yourself". Maybe he needs that.

  • is it easy to become a dispatcher?

    Jump

  • Won't surprise me if that's already automated.

  • is it easy to become a dispatcher?

    Jump

  • Um what kind of dispatcher? Like 911 or what?

  • Wikimedia staff in the US have formed a trade union, but the foundation has refused to recognise it

    Jump

  • Yeah the founder is or was into that. Thus the idea that Wikipedia's editorial decisions could essentially be made by algorithms. I have some jokes about it but will spare you for now. The WMF itself is just another tech company with some nonprofit virtue vibes though. Expect a typical campaign.

  • Is life really as awful as politics and the Internet make it out to be?

    Jump

  • I'd say if you are able to maintain steady income, friends, interests, don't live in an active war zone or other disaster area, and have a few other lucky circumstances like that, you can (at least for now) have a reasonably happy existence. You don't have to ignore or deny the terrible stuff going on elsewhere. There have always been wars and poverty happening, but it's ok to respond more to your sense of your immediate surroundings (where things are going ok) than to your sense of bad stuff happening in remote times and places.

    The internet and political news reminds us that bad times may be coming for everyone due to global warming, AI propaganda from encroaching dictatorships, and all that stuff. Don't ignore that. Oppose it if you can. But, don't let it overcome your hopefully-reasonably-good existence in the present.

    Certainly, if spending all day on the internet makes you neurotic, then try to cut back on it. Some people choose to eliminate it completely from their lives, at least for a while. I haven't done that so don't know if it helps.

  • T-Mobile Is Down. If your phone isn't working, it's not just you: T-Mobile is down.

    Jump

  • Up for me right now, northern California.

  • Do you expect, or even hope to die alone?

    Jump

  • Old joke: I hope to die peacefully in my sleep like my grandfather, not screaming in terror like his passengers.

  • Removed Deleted

    Permanently Deleted

    Jump

  • Scientists Build the First 3D Cloak That Makes Objects Invisible To Heat

    Jump

  • This sounds pretty neat. It sounds like it uses transformation optics, which I've just been reading about.

    You know how a lens bends light, i.e. a light ray enters the lens and travels at a different angle that depends on the lens's index of refraction? Now suppose the index of refraction changes continuously, like air's index of refraction changes with temperature. So if you have warm air near the ground that gets cooler with height, light can actually travel through it by a curved path. That's how mirages happen.

    Now suppose you can somehow control the index of refraction in a volume of material. So you can imagine an "invisibility cloak" where light coming in from behind you gets steered around you and comes out in front of you. People looking straight at you see the light from in back of you so it's like you're invisible. This doesn't seem technologically doable with visible light (at least for now) but it's been demonstrated in the lab at microwave frequencies, with small objects like a few centimeters. It's really cool how the math works.

    This sounds like they've done something similar with heat. They use a 3D printed heat sink with a special shape to make the temperature in front of something look like the temperature behind the thing, masking the heat source in the middle.

    If you look up "transformation optics" there's a lot of stuff about invisibility cloaks, maybe because there's military funding in the field and the army wants to make stuff invisible, lol. But the math and physics are interesting even if the applications are overstated. It turns out that the math used in studying how the materials curve light is similar to the math of general relativity, which is about how gravity bends light near stars and black holes. So that also leads to a bunch of breathless headlines about making black holes in the laboratory. It's not 100% bogus (they do get some cool effects) but yeah, of course it's an analogy, not an actual black hole.

    Here's a TED talk by one of the guys involved with this. I haven't watched it yet but maybe I will. I saw another youtube lecture by him and it was interesting. https://www.youtube.com/watch?v=v9d-EBgro50

  • What's something you know about philosophy?

    Jump

  • Time flies like an arrow. Fruit flies like an orange.

  • What's something you know about philosophy?

    Jump