starred/ambar
1
0
Fork 0
mirror of https://github.com/RD17/ambar.git synced 2026-04-25 15:35:49 +03:00
Code Issues 2 Projects Releases 3 Packages Wiki Activity

[GH-ISSUE #33] Feature request: RSS feed crawler #33

New issue
Closed
opened 2026-02-27 15:54:34 +03:00 by kerem · 4 comments
kerem commented 2026-02-27 15:54:34 +03:00
Owner
Copy link

Originally created by @jenkin on GitHub (May 11, 2017).
Original GitHub issue: https://github.com/RD17/ambar/issues/33

When items contain enclosures linking to available documents.

Originally created by @jenkin on GitHub (May 11, 2017). Original GitHub issue: https://github.com/RD17/ambar/issues/33 When items contain [enclosures](http://cyber.harvard.edu/rss/rss.html#ltenclosuregtSubelementOfLtitemgt) linking to available documents.
kerem 2026-02-27 15:54:34 +03:00
kerem commented 2026-02-27 15:54:35 +03:00
Author
Owner
Copy link

@sochix commented on GitHub (May 28, 2017):

Hi @jenkin . Can you please describe this crawler a bit? Should it fetch all linked documents? And if the document an html page should it fetch it too?

<!-- gh-comment-id:304493433 --> @sochix commented on GitHub (May 28, 2017): Hi @jenkin . Can you please describe this crawler a bit? Should it fetch all linked documents? And if the document an html page should it fetch it too?
kerem commented 2026-02-27 15:54:35 +03:00
Author
Owner
Copy link

@jenkin commented on GitHub (May 30, 2017):

RSS specifications require the mime type as attribute of enclosure tag: "It has three required attributes: url says where the enclosure is located, length says how big it is in bytes, and type says what its type is, a standard MIME type." So the crawler should fetch only enclosures with a supported (by Ambar) mime type (ie. application/pdf). Other types, such as mp3 in podcast feeds, will be ignored.

<!-- gh-comment-id:304795499 --> @jenkin commented on GitHub (May 30, 2017): RSS specifications require the mime type as attribute of [enclosure tag](http://cyber.harvard.edu/rss/rss.html#ltenclosuregtSubelementOfLtitemgt): "It has three required attributes: url says where the enclosure is located, length says how big it is in bytes, and type says what its type is, a standard MIME type." So the crawler should fetch only enclosures with a supported (by Ambar) mime type (ie. application/pdf). Other types, such as mp3 in podcast feeds, will be ignored.
kerem commented 2026-02-27 15:54:36 +03:00
Author
Owner
Copy link

@sochix commented on GitHub (May 30, 2017):

@jenkin got it! Can you please also describe use case for the RSS crawler? What kind of data you want to crawl with it? Newsletters or what?

<!-- gh-comment-id:304800553 --> @sochix commented on GitHub (May 30, 2017): @jenkin got it! Can you please also describe use case for the RSS crawler? What kind of data you want to crawl with it? Newsletters or what?
kerem commented 2026-02-27 15:54:36 +03:00
Author
Owner
Copy link

@sochix commented on GitHub (Apr 19, 2018):

Check our support options

<!-- gh-comment-id:382659375 --> @sochix commented on GitHub (Apr 19, 2018): [Check our support options](https://github.com/RD17/ambar#support)
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
v2.1.18
v2.1.8
v2.0.0rc
Labels
Clear labels   
$$ Paid Support

   
bug

   
bug

   
enhancement

   
help wanted

   
invalid

   
pull-request

Mirrored from GitHub Pull Request

  
question

   
question

   
wontfix
No labels
$$ Paid Support
bug
bug
enhancement
help wanted
invalid
pull-request
question
question
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/ambar#33
No description provided.