Skip Navigation

InitialsDiceBearhttps://github.com/dicebear/dicebearhttps://creativecommons.org/publicdomain/zero/1.0/„Initials” (https://github.com/dicebear/dicebear) by „DiceBear”, licensed under „CC0 1.0” (https://creativecommons.org/publicdomain/zero/1.0/)C

cll7793

@ cll7793 @lemmy.world

Posts
12
Comments
16
Joined
3 yr. ago

Hi, I like to learn about what resources are out there on the internet. I hope you have found my posts useful!

  • What are some tricks to efficiently search for information on the internet?

    Jump

  • What are some tricks to efficiently search for information on the internet?

    Jump

  • Super useful plugin! You can also subscribe to lists that block SEO/AI generated websites. Now only if there was a whitelist plugin that places forums higher up

  • What are some tricks to efficiently search for information on the internet?

    Jump

  • Someday this will be possible when an open source search engine comes around.

  • What are some tricks to efficiently search for information on the internet?

    Jump

  • I noticed some of the best resources from the past are unfindable from any search engine. For example some science youtube channels which offer amazing quality content seem to be unfindable. They are replaced with other channels that try to clickbait their way to the top. The same can be said with websites that SEO as much as they can. The highest quality resources are also often in the least quantity. A form of quantity > quality is favored and amplified and sometimes even censored. (Anna's archive)

  • What are some tricks to efficiently search for information on the internet?

    Jump

  • It's quite sad that we are now at a point where we are forced to make our own search engines from scratch. Search engines are hard! Google's original search algorithm (about 2 decades ago) was quite amazing. You were able to give vague search terms and yet still find the answer you wanted. The secret sauce was ranking based on relevance to the search query. I'm not aware of any guides/projects on search engines. I wish there was a good way I could search for this. (The irony!) But a great starting resource is this series on networks from wikipedia. (https://en.wikipedia.org/wiki/Network_theory)

    Some random tips:

    • The main goal of any search engine should be to minimize the number of times a user returns to the ranking page to click on a new link. Big tech should be doing this anyways but they have other goals.
    • The main metadata database needs to topologically connect you to any part of the internet. (https://en.wikipedia.org/wiki/Graph_theory) Think of it as a hub/portal giving you general directions, but doesn't tell you exactly where you should be heading. The ideal solution is to download everything from the internet and query each result for relevance to a search query individually, but this is intractable. Instead you have to group the internet into graphs and sub graphs - STEM, Social, Forums, E-commerce etc. Hyperlinks offer an objective way to calculate connections between websites. For example Lemmy.world <-> Wikipedia.org. The weight of these connections gives you a way to guide a traversal algorithm during search. Semantic analysis of some form allows you to find more efficient ways to draw connections making your search more efficient.
    • The most powerful way to find connection/relevance to a search term is with transformers and their attention mechanism. For example if the search query is "Open source search engine", the attention heatmap would be on groups of websites subjects like Forums, Q&A, Programming, Network Science, etc. There would also be a negative heatmap for topics like Cooking, Sports, Entertainment, etc. From there you want to load up recursively metadata for websites. For example for Lemmy it would be the title of all posts (and maybe their top comments). If it fits, load as much of this as you can into a transformer and calculate the heatmap relative to the search query. Again you are not using the transformer to generate answers. This is a bad idea. Instead you are using it to rank search results in terms of relevance/attention, what the transformer is fundamentally designed for.

    As a side note, you are able to tune your model to your own search preferences with little data. You are also able to exchange computation time for search quality! This is amazing. If computation is a concern, traditional traversal algorithms and basic relevance/ranking algorithms work too but at the cost of more engineering.

    I hope this sorta helps, if you have any other question feel free to ask! The future of search will likely be self-hosted as conflicts of interest within current search engine providers degrades the quality to the point where they are unusable.

  • What are some tricks to efficiently search for information on the internet?

    Jump

  • Finding the balance between what to keep to index is hard! The attention mechanism in transformers should be pretty good at ranking results. The idea is to feed into context titles, top answers, etc in bulk along with a search query. The attention heatmap relative to the search gives you a general rank for how good each result is. Ironically enough, this is probably the most powerful indexer, yet no big tech uses it and instead has the model generate answers instead of ranking them. The best part is, this system is tunable and can be adjusted to user preference with little data. The overall goal should be to minimize the number of results a user checks. (This should be what other engines are doing in the first place)

  • Ask Lemmy @lemmy.world
    cll7793 @lemmy.world

    What are some tricks to efficiently search for information on the internet?

  • Ask Lemmy @lemmy.world
    cll7793 @lemmy.world

    How do you get the best prices? (General Finance)

  • What are some powerful open source projects everyone should know?

    Jump

  • Thank you! I didn't know about them!!

  • What are some powerful open source projects everyone should know?

    Jump

  • I know right? Open source hardware has so many potential benefits over commercial. Significantly decreased price, privacy, good documentation, right to repair, no conflict of interest and potentially one day performance. Imagine we have engineers from across the world improving a single computer chip design, generator design, solar panel fabrication process, or maybe even perhaps an open source fusion reactor blueprint someday in the next 20 years (pun intended).

    I'm seriously considering starting something like this myself. Open source blueprints for power generation/energy storage (regular batteries, thermal sand resevior based batteries, hydro power generation), water filtration, machine tools for fabricating anything, CNC machine, plasma cutters, hand tools, etc. Basically everything you could need to live Open Source.

    The problem as always is getting enough designers, engineers, and volunteers.

  • What are some powerful open source projects everyone should know?

    Jump

  • Oh? I didn't know that. They seemed like a good organization to me too. Open source hardware is quite lacking compared to the software side so I hope they succeed.

  • What forums/social media sites do you recommend?

    Jump

  • Found a nice list of resources for RSS. The internet really does feel like an echo-chamber now! There is so much noise to signal. Hopefully this post helps reduce the echoing a tad. I would love if you shared some tips for getting started with RSS if that's alright!

  • What forums/social media sites do you recommend?

    Jump

  • What are some powerful open source projects everyone should know?

    Jump

  • Sorry didn’t mean to cause any trouble. I collect and share internet resources with others. If you want to verify this for yourself, my post history has questions similar to this one. I removed the image to make this post more ‘generic’. I am genuinely trying to share resources. My apologies if it came across as advertising.

  • What are some powerful open source projects everyone should know?

    Jump

  • What are some powerful open source projects everyone should know?

    Jump

  • Thank you, great resource! It's very sad to see the feds going after libraries with the recent attacks against libgen (And potentially Internet Archive). Hopefully these libraries truly will be preserved forever. Information belongs to everyone.

    In his ruling, Judge McMahon wrote, 'The plaintiffs (the publishers) have suffered irreparable harm as a result of the defendants' (LibGen operators') unlawful conduct and will continue to suffer harm if the defendants are permitted to continue operating LibGen,' and said LibGen's operation should be stopped.

    Source: https://gigazine.net/gsc_news/en/20240927-libgen-must-pay-30m-to-publishers/

  • Ask Lemmy @lemmy.world
    cll7793 @lemmy.world

    What are some powerful open source projects everyone should know?

  • Ask Lemmy @lemmy.world
    cll7793 @lemmy.world

    What forums/social media sites do you recommend?

    en.wikipedia.org /wiki/List_of_Internet_forums
  • Ask Lemmy @lemmy.world
    cll7793 @lemmy.world

    What are some financial life-hacks everyone should know?

  • Ask Lemmy @lemmy.world
    cll7793 @lemmy.world

    What free things online should everyone take advantage of?

  • Ask Lemmy @lemmy.world
    cll7793 @lemmy.world

    What social media platforms do you recommend?

  • Ask Lemmy @lemmy.world
    cll7793 @lemmy.world

    What are some must watch Documentaries?

  • Ask Lemmy @lemmy.world
    cll7793 @lemmy.world

    What are the best RSS feeds on the Internet?

  • Ask Lemmy @lemmy.world
    cll7793 @lemmy.world

    What are some useful links everyone should know?

  • What are some quality YouTube channels that you recommend?

    Jump

  • Led Fridman

  • Ask Lemmy @lemmy.world
    cll7793 @lemmy.world

    What are the best sources of information on the Internet?