Web Scraping Services

The Vendor is required to provide web scraped data from state websites that are in a structured format, for use by agency for statistical and research purposes.
- Agency is only interested in acquiring data, and not in purchasing software or a license to use an established software as a service (SAAS) solution.
- Services include:
• Work with the project authority to establish the details of the variables collected from each website and their associated format. 
• The project authority may request additional variables not specified in section SW 4.2 (product availability flags, promotional pricing data, etc.).
• Work with the project authority to establish a delivery schedule for the structured data files in batches based on a pre-determined schedule defined by the contractor and agency.
• Complete the scrape of each website or portion of each website as agreed upon with agency on a weekly or monthly basis, or ad hoc as per agency requirements.
• Transmit the collected data through agency secure electronic file transfer (eft) service at current minimum of once per week. 
• The data must be transmitted no more than one week after it has been collected.
• Provide an up-to-date record layout for each data file transmitted.
• Notify the project authority of any issues or delays with data transmission within one business day.
• Resolve any issues that prevent the collection of data due to a website overhaul within fourteen (14) calendar days.
• Resolve any other issues that prevent the collection of data within seven (7) calendar days.
• Setup new website scrapes within thirty (30) calendar days of agency requesting a new website to be scrapped.
• Cancel and stop billing for website scrapes/maintenance fees within thirty (30) calendar days of agency confirming the data is no longer required.
- Contract Period/Term: 1 year