IT Job Trend Mining and Analysis

Abstract

Data mining is the hot term in this few years in the information technology industry. More and more people want to use the existing huge data to find out the business value, helping for the research or predict the future. Sine ebay needs to 40PB of their historical data. Facebook handles 50PB photos in their database. Walmart handles more than 1 million buying transactions every hour. It seems that there is huge of historical data, so how to use those resource to convert into values and knowledge. In this project, it brings the research on the IT industry to find out United States and Hong Kong job market. It can discover the difference between the technologies needed, the IT development process and career path in this two district. To implement this project, distributed parallel computing system will be used to achieve the big data processing. We try to break down the limitation of traditional data analysis methods and data warehouse problem when facing the batch data processing and also discover the knowledge through the data mining techniques from the unstructured data and information.