Generated HTML Output: A Preview of What the Code Produces<\/h1>\n\n\n\n
The code generates an in-depth profile of any stock specified. In this example, we take a look at Google. The output is a structured HTML file that includes a variety of information sections. First, it provides basic details such as the company’s name, ticker symbol, and the date of the most recent update. Next, it lists essential contact information, including the company’s web URL, phone number, and physical address. The sector and industry in which the company operates, as well as the number of full-time employees, are also displayed. The output then delves into valuation metrics like P\/E ratios, enterprise value, and other financial ratios. Finally, it highlights key financial data points, such as market capitalization, EBITDA, and various earnings and revenue estimates. Overall, the code produces a multi-faceted profile that serves as a valuable resource for anyone looking to understand Alphabet Inc’s business and financial standing in great detail.<\/p>\n\n\n\n This plot visualizes the historical and predicted adjusted close prices for a particular asset, utilizing the Facebook Prophet algorithm for the forecast. The x-axis represents the timeline, extending from the earliest available historical data to one year into the future. The y-axis shows the adjusted close prices. Historical prices are plotted as solid purple lines. The predicted value is a solid green line with blue dotted lines as the high and low predictions. This predictive model leverages the power of Facebook Prophet to analyze seasonal and trend components in the historical data, providing a one-year outlook on potential price movements.<\/p>\n\n\n\n Forecasting Future Financial Ratios with Facebook Prophet: A One-Year Outlook<\/strong><\/p>\n\n\n\n Financial ratios are key indicators of a company’s financial health and performance. They offer crucial insights into various aspects like profitability, liquidity, and valuation. In this section, we’ll leverage the predictive power of Facebook Prophet to forecast these ratios over the next year.<\/p>\n\n\n\n These plots present a comprehensive visualization of key earnings and profitability metrics extracted from the income statement, including Net Income, Operating Income, Total Revenue, Gross Profit, and Free Cash Flow. The financial numbers are then forward projected using Facebook Prophet. These predictions not only display the expected trends for the next year but also encompass upper and lower bounds, providing a range of possible outcomes. This integration of historical data with advanced forecasting techniques offers a nuanced understanding of the company’s financial trajectory.<\/p>\n\n\n\n These plots offer an in-depth analysis of crucial balance sheet metrics such as Total Assets, Total Stockholder Equity, Retained Earnings, Long-Term Debt, and Total Liabilities. Leveraging the predictive power of Facebook Prophet, these key financial indicators are forecasted into the future. The projections not only illustrate the expected financial posture for the forthcoming year but also include upper and lower prediction intervals, giving a full spectrum of potential financial scenarios. By melding past balance sheet data with sophisticated predictive modeling, the plots provide a multi-dimensional view of the company’s expected financial stability and risk profile.<\/p>\n\n\n\n These plots dive into essential cash flow metrics, specifically Total Cash from Operating Activities, Capital Expenditures, and Dividends Paid. These key figures are also extended into the future using Facebook Prophet’s forecasting capabilities. The resulting predictions not only map out the anticipated cash flow movements for the next year but are also bracketed by upper and lower confidence intervals, presenting a comprehensive range of financial possibilities. These plots offer a well-rounded perspective on the company’s future liquidity and capital allocation strategies.<\/p>\n\n\n\n In addition to the key financial metrics, the analysis delves into a diverse set of other metrics, plotting each meticulously. These metrics are categorized as follows:<\/p>\n\n\n\n The code includes several helper functions, such as:<\/p>\n\n\n\n The core of this code is the <\/p>\n\n\n\n <\/p>\n","protected":false},"excerpt":{"rendered":" I started off wanting to analyze the sustainability of mortgage buydowns by home builders. But as it happens, my ADHD had other plans. Three weeks later, here we…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-5099","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"\n![\"\"](\"https:\/\/new.jeremywhittaker.com\/wp-content\/uploads\/2023\/11\/Google_cut_sheet-1024x460.png\")
<\/a><\/figure>\n\n\n\nForecasting Future Adjusted Close Prices with Facebook Prophet: A One-Year Outlook<\/h1>\n\n\n\n
![\"\"](\"https:\/\/new.jeremywhittaker.com\/wp-content\/uploads\/2023\/11\/image-1-1024x161.png\")
<\/a><\/figure>\n\n\n\n![\"\"](\"https:\/\/new.jeremywhittaker.com\/wp-content\/uploads\/2023\/11\/goog_pe_ratio-1024x208.png\")
<\/a><\/figure>\n\n\n\n![\"\"](\"https:\/\/new.jeremywhittaker.com\/wp-content\/uploads\/2023\/11\/goog_pb_ratio-1024x243.png\")
<\/a><\/figure>\n\n\n\n![\"\"](\"https:\/\/new.jeremywhittaker.com\/wp-content\/uploads\/2023\/11\/goog_roe-1024x219.png\")
<\/a><\/figure>\n\n\n\n![\"\"](\"https:\/\/new.jeremywhittaker.com\/wp-content\/uploads\/2023\/11\/goog_earnings_yield-1024x197.png\")
<\/a><\/figure>\n\n\n\n![\"\"](\"https:\/\/new.jeremywhittaker.com\/wp-content\/uploads\/2023\/11\/goog_dividend_yield-1024x186.png\")
<\/a><\/figure>\n\n\n\n![\"\"](\"https:\/\/new.jeremywhittaker.com\/wp-content\/uploads\/2023\/11\/goog_current_ratio-1024x188.png\")
<\/a><\/figure>\n\n\n\nForecasting Future Earnings and Profitability Metrics from the Income Statement with Facebook Prophet: A One-Year Outlook<\/h1>\n\n\n\n
![\"\"](\"https:\/\/new.jeremywhittaker.com\/wp-content\/uploads\/2023\/11\/image-2-1024x976.png\")
<\/a><\/figure>\n\n\n\nForecasting Future Balance Sheet Metrics with Facebook Prophet: A One-Year Outlook<\/h1>\n\n\n\n
![\"\"](\"https:\/\/new.jeremywhittaker.com\/wp-content\/uploads\/2023\/11\/image-3-988x1024.png\")
<\/a><\/figure>\n\n\n\nForecasting Future Cash Flow Metrics with Facebook Prophet: A One-Year Outlook<\/h1>\n\n\n\n
![\"\"](\"https:\/\/new.jeremywhittaker.com\/wp-content\/uploads\/2023\/11\/image-4-1024x646.png\")
<\/a><\/figure>\n\n\n\nOther Financial Metrics<\/h1>\n\n\n\n
Efficiency and Activity Metrics from Income Statement:<\/h2>\n\n\n\n
\n
Stockholder’s Equity and Capital Structure:<\/h2>\n\n\n\n\n
Additional Important Metrics:<\/h2>\n\n\n\n\n
Less Critical Metrics:<\/h2>\n\n\n\n
\n
Other Metrics:<\/h2>\n\n\n\n
\n
Code Overview<\/h1>\n\n\n\n
Prerequisites<\/h1>\n\n\n\n
\n
Helper Functions<\/h1>\n\n\n\n
\n
calculate_mae<\/code>: Calculates the Mean Absolute Error between the actual and predicted data.<\/li>\n\n\n\naccess_hdf5_with_retries<\/code>: Attempts to read an HDF5 file, retrying up to a specified number of times.<\/li>\n\n\n\nread_general_info<\/code>: Reads general company information from an HDF5 file.<\/li>\n\n\n\ninfer_dtype<\/code>: Infers the data type of a Pandas Series.<\/li>\n\n\n\nfetch_data_for_symbol_from_multiple_h5<\/code>: Fetches data for a specific symbol from multiple HDF5 files.<\/li>\n\n\n\nget_company_name<\/code>: Gets the company name for a given symbol from an HDF5 file.<\/li>\n\n\n\nadd_content_before_plot<\/code>: Adds additional content before the plot in an HTML file.<\/li>\n<\/ul>\n\n\n\nForecasting Function<\/h1>\n\n\n\n
forecast_with_multiple_metrics<\/code> function, which:<\/p>\n\n\n\n\n
def forecast_with_multiple_metrics(df: pd.DataFrame, periods: int = 4, save_dir: str = \"plots\/\", use_all_data: bool = True, eod_price_data: str = 'eod_price_data.h5'):\n ...\n<\/code><\/pre>\n\n\n\nCode<\/h1>\n\n\n\n
import pandas as pd\nfrom pathlib import Path\nimport logging\nimport os\nfrom plotly.subplots import make_subplots\nfrom prophet import Prophet\nimport plotly.graph_objects as go\nfrom collections import Counter\nfrom pandas.api.types import is_numeric_dtype\nfrom bs4 import BeautifulSoup\nimport time\nfrom sklearn.metrics import mean_absolute_error\nfrom math import log10\n\n\nfrom dictionaries_and_lists import metric_definitions, reordered_columns, homebuilders, sp500, companies_with_treasuries, largest_banks, percentage_metrics\n\ndata_dir = '\/home\/shared\/algos\/data\/'\nplots_dir = '\/home\/shared\/algos\/eodhd_data\/plots\/'\n\nlogging.basicConfig(level=logging.DEBUG)\nlogging.basicConfig(level=logging.INFO)\n\n# Configure Pandas to display all columns\npd.set_option('display.max_columns', None)\npd.set_option('display.expand_frame_repr', False)\n\nsymbol_exchange_map = {}\n\n\ndef wrap_text(text, max_length):\n \"\"\"\n Wraps text to a new line at the nearest whitespace of the max_length.\n \"\"\"\n wrapped_lines = []\n while len(text) > max_length:\n # Find nearest whitespace of the max_length\n split_index = text.rfind(' ', 0, max_length + 1)\n if split_index == -1: # No whitespace found, force split\n split_index = max_length\n wrapped_lines.append(text[:split_index])\n text = text[split_index:].lstrip()\n wrapped_lines.append(text)\n return '<br>'.join(wrapped_lines)\n\ndef format_large_number(num):\n print('Attempting to plot large number')\n if num < 1_000:\n return str(num)\n magnitude = int(log10(num) \/\/ 3)\n value = num \/ (10 ** (3 * magnitude))\n return f\"{value:.2f}{' KMBT'[magnitude]}\"\n\ndef format_percentage(data):\n try:\n # Remove any non-numeric characters like commas and percentage signs\n if isinstance(data, str):\n data = data.replace(',', '').replace('%', '').strip()\n # Convert to float and format as a percentage\n return \"{:.2%}\".format(float(data))\n except ValueError as e:\n print(f\"ValueError: Could not convert {data} to a percentage.\")\n return data # Return the original data if it cannot be converted\n\n\ndef calculate_mae(actual, predicted):\n return mean_absolute_error(actual, predicted)\n\ndef access_hdf5_with_retries(hdf5_file_path, mode, max_retries=3, sleep_duration=5):\n retries = 0\n while retries < max_retries:\n try:\n with pd.HDFStore(hdf5_file_path, mode) as store:\n return store.keys() # You can customize this part to return what you need\n break # If successful, exit the while loop\n except Exception as e: # Replace Exception with a more specific exception if possible\n if retries < max_retries - 1:\n logging.info(f\"An exception occurred while reading {hdf5_file_path}: {e}\")\n logging.info(f\"Retrying in {sleep_duration} seconds...\")\n time.sleep(sleep_duration)\n retries += 1\n else:\n logging.error(f\"Max retries reached. Could not read {hdf5_file_path}. Exiting...\")\n raise\n\ndef read_general_info(symbol, h5_general_path, max_retries=3, sleep_duration=5):\n retries = 0\n while retries < max_retries:\n try:\n with pd.HDFStore(h5_general_path, 'r') as store:\n key = f'\/{symbol}'\n if key in store.keys():\n general_info = store.get(key)\n info_dict = dict(zip(general_info['SubCategory'], general_info['Data']))\n sector = info_dict.get('Sector', 'N\/A')\n industry = info_dict.get('Industry', 'N\/A')\n description = info_dict.get('Description', 'N\/A')\n full_time_employees = info_dict.get('FullTimeEmployees', 'N\/A')\n updated_at = info_dict.get('UpdatedAt', 'N\/A')\n web_url = info_dict.get('WebURL', 'N\/A')\n phone = info_dict.get('Phone', 'N\/A')\n address = info_dict.get('Address', 'N\/A')\n name = info_dict.get('Name', 'N\/A')\n exchange = info_dict.get('Exchange', 'N\/A')\n return sector, industry, description, full_time_employees, updated_at, web_url, phone, address, name, exchange\n else:\n return ['N\/A']*10\n break\n except Exception as e:\n if retries < max_retries - 1:\n print(f\"An exception occurred while reading {h5_general_path}: {e}\")\n print(f\"Retrying in {sleep_duration} seconds...\")\n time.sleep(sleep_duration)\n retries += 1\n else:\n print(f\"Max retries reached. Could not read {h5_general_path}. Exiting...\")\n raise\n\n\n\ndef infer_dtype(series):\n sample = series.dropna().head(100) # Sample the first 100 non-null rows\n\n # Debugging line\n print(f\"Sample for {series.name}: {sample}\")\n\n if sample.empty:\n return None\n\n if all(sample.apply(lambda x: isinstance(x, str))):\n return 'object'\n\n if is_numeric_dtype(sample) and all(sample.apply(lambda x: x == int(x))):\n return 'int64'\n\n if is_numeric_dtype(sample):\n return 'float64'\n return None\n\n\ndef fetch_data_for_symbol_from_multiple_h5(h5_filepaths, symbol, max_retries=3, sleep_duration=5):\n combined_data = None\n common_keys = ['Symbol', 'date', 'filing_date', 'currency_symbol']\n\n for h5_filepath in h5_filepaths:\n h5_path = Path(h5_filepath)\n if not h5_path.exists():\n logging.info(f\"The file {h5_filepath} does not exist.\")\n continue\n\n retries = 0\n while retries < max_retries:\n try:\n with pd.HDFStore(h5_filepath, 'r') as store:\n if f\"\/{symbol}\" in store.keys():\n logging.info(f\"Symbol {symbol} found in {h5_filepath}.\")\n symbol_data = store.select(symbol)\n\n if 'date' not in symbol_data.columns and 'Date' not in symbol_data.columns:\n if isinstance(symbol_data.index, pd.DatetimeIndex):\n symbol_data.reset_index(inplace=True)\n symbol_data.rename(columns={'index': 'date'}, inplace=True)\n logging.info(\n f\"'date' and 'Date' columns not found, but datetime index exists in {h5_filepath} for symbol {symbol}.\")\n symbol_data = symbol_data.reset_index().rename(columns={'index': 'date'})\n else:\n logging.warning(\n f\"'date' and 'Date' columns and datetime index not found in {h5_filepath} for symbol {symbol}.\")\n break\n else:\n # Ensure the date column is standardized to 'date'\n if 'Date' in symbol_data.columns:\n symbol_data.rename(columns={'Date': 'date'}, inplace=True)\n\n\n # Rename 'netIncome' if it's present in the dataframe\n if 'netIncome' in symbol_data.columns:\n # Extract the type of financial statement from the filename\n parts = h5_path.stem.split('_')\n statement_type = parts[1]\n metric_name = parts[2]\n\n statement_readable = {\n \"Cash\": \"Cash Flow from Operating Activities\",\n \"Income\": \"netIncome\",\n }.get(statement_type, statement_type)\n\n metric_title = f\"{metric_name} ({statement_readable})\"\n\n # Ensure statement_type is one of the reports where 'netIncome' appears before renaming\n if statement_type in [\"Income\", \"Cash\"]:\n symbol_data.rename(columns={'netIncome': metric_title}, inplace=True)\n\n if combined_data is None:\n combined_data = symbol_data\n else:\n # Dynamically adjust common keys based on available columns\n keys_for_merge = [key for key in common_keys if\n key in symbol_data.columns and key in combined_data.columns]\n combined_data = pd.merge(combined_data, symbol_data, on=keys_for_merge, how='outer')\n break\n\n else:\n logging.info(f\"Symbol {symbol} not found in {h5_filepath}.\")\n break\n\n except Exception as e:\n if retries < max_retries - 1:\n logging.info(f\"An exception occurred while reading {h5_filepath}: {e}\")\n logging.info(f\"Retrying in {sleep_duration} seconds...\")\n time.sleep(sleep_duration)\n retries += 1\n else:\n logging.info(f\"Max retries reached. Could not read {h5_filepath}. Exiting...\")\n raise\n\n if combined_data is None:\n logging.info(f\"Symbol {symbol} not found in any of the HDF5 files.\")\n return None\n else:\n for col in combined_data.select_dtypes(include=['object']).columns:\n combined_data[col] = pd.to_numeric(combined_data[col], errors='ignore')\n\n if 'date' in combined_data.columns:\n combined_data.sort_values(by='date', inplace=True)\n else:\n logging.info(f\"'date' column not found in combined data for symbol {symbol}. Sorting by index instead.\")\n combined_data.sort_index(inplace=True)\n logging.info(f\"{symbol}: successfully combined all HDF5 files.\")\n return combined_data\n\n\n\ndef get_company_name(symbol, h5_file_path, country_code='US'):\n try:\n # Read the DataFrame for the entire country from the HDF5 file\n symbols_df = pd.read_hdf(h5_file_path, key=f'\/{country_code}')\n\n # Filter by the specific symbol to get the company name\n company_name_row = symbols_df[symbols_df['Code'] == symbol]\n if not company_name_row.empty:\n return company_name_row['Name'].iloc[0]\n else:\n logging.error(f\"No data for symbol {symbol} in the DataFrame.\")\n return \"Unknown\"\n except KeyError:\n logging.error(f\"No object named {country_code} in the file {h5_file_path}\")\n return \"Unknown\"\n\n\ndef add_content_before_plot(symbol, max_retries=3, sleep_duration=5):\n target_filename = f\"{symbol}_quarterly.html\"\n html_file_path = os.path.join(plots_dir, target_filename)\n\n if not os.path.exists(html_file_path):\n print(f\"HTML file for symbol {symbol} not found.\")\n return\n\n # Fetch additional info from General.h5\n exchange = symbol_exchange_map.get(symbol, 'Other') # Default to 'Other' if exchange is not found\n h5_general_path = os.path.join(data_dir, f\"{exchange}_General.h5\")\n\n sector, industry, description, full_time_employees, updated_at, web_url, phone, address, name, exchange = read_general_info(symbol, h5_general_path, max_retries, sleep_duration)\n\n # Prepare new HTML content\n new_html_content = f\"\"\"\n <h1>{symbol} - {name} - {exchange}<\/h1>\n <p>Updated At: {updated_at}<\/p>\n <p><strong style='font-size: larger;'>WebURL:<\/strong> <a href='{web_url}' target='_blank'>{web_url}<\/a>\n <strong style='font-size: larger;'>Phone:<\/strong> {phone}\n <strong style='font-size: larger;'>Address:<\/strong> {address}<br>\n <strong style='font-size: larger;'>Sector:<\/strong> {sector}\n <strong style='font-size: larger;'>Industry:<\/strong> {industry}<br>\n <strong style='font-size: larger;'>Full Time Employees:<\/strong> {full_time_employees}<\/p><br>\n <strong style='font-size: larger;'>Description:<\/strong> {description}<br><br>\n \"\"\"\n\n # Read valuation.h5 data for the symbol\n h5_valuation_path = os.path.join(data_dir, f\"{exchange}_Valuation.h5\")\n\n retries = 0\n valuation_data = None\n while retries < max_retries:\n try:\n with pd.HDFStore(h5_valuation_path, 'r') as store:\n if symbol in store:\n valuation_data = store.get(symbol)\n else:\n valuation_data = pd.DataFrame(columns=['SubCategory', 'Data'])\n print(f\"The symbol {symbol} valuation data does not exist in {h5_valuation_path}\")\n break\n except Exception as e:\n if retries < max_retries - 1:\n print(f\"An exception occurred while reading {h5_valuation_path}: {e}\")\n print(f\"Retrying in {sleep_duration} seconds...\")\n time.sleep(sleep_duration)\n retries += 1\n else:\n print(f\"Max retries reached. Could not read {h5_valuation_path}. Exiting...\")\n raise\n\n # Add valuation_data to new_html_content\n valuation_html = \"<h2>Valuation<\/h2><div style='display: flex; flex-wrap: wrap;'>\"\n if valuation_data is not None:\n print(f\"valuation_data: {valuation_data}\")\n valuation_data['Data'] = pd.to_numeric(valuation_data['Data'], errors='coerce')\n\n for index, row in valuation_data.iterrows():\n data = row['Data']\n if pd.notnull(data): # Check if 'Data' is not NaN\n if index in percentage_metrics:\n formatted_data = \"{:.2%}\".format(float(data))\n elif isinstance(data, (int, float)) and abs(data) >= 1_000: # Large numbers\n formatted_data = format_large_number(data)\n elif isinstance(data, (int, float)):\n print('formatting numbers with commas in Valuation')\n print(f\"data: {data}, type: {type(data)}\")\n formatted_data = f\"{data:,.2f}\" # For other numbers, just format with commas\n else:\n formatted_data = data # For non-numeric data, leave as is\n else:\n # If data is NaN or non-numeric, leave as is\n formatted_data = row['Data']\n\n valuation_html += f\"<div style='flex: 0 0 calc(33.333% - 10px); margin-right: 10px; margin-bottom: 10px;'>\"\n valuation_html += f\"<strong>{row['SubCategory']}<\/strong>: {formatted_data}<\/div>\"\n valuation_html += \"<\/div>\"\n\n new_html_content += valuation_html # Append valuation data to new_html_content\n\n\n\n # Read highlights.h5 data for the symbol\n h5_highlights_path = os.path.join(data_dir, f\"{exchange}_Highlights.h5\")\n retries = 0\n while retries < max_retries:\n try:\n with pd.HDFStore(h5_highlights_path, 'r') as store:\n if symbol in store:\n highlights_data = store.get(symbol)\n else:\n highlights_data = pd.DataFrame(columns=['SubCategory', 'Data'])\n print(f\"The symbol {symbol} highlights data does not exist in {h5_highlights_path}\")\n break\n except Exception as e:\n if retries < max_retries - 1:\n print(f\"An exception occurred while reading {h5_highlights_path}: {e}\")\n print(f\"Retrying in {sleep_duration} seconds...\")\n time.sleep(sleep_duration)\n retries += 1\n else:\n print(f\"Max retries reached. Could not read {h5_highlights_path}. Exiting...\")\n raise\n\n # Add title before highlights\n new_html_content += \"<h2>Highlights<\/h2>\"\n # Add highlights_data to new_html_content\n highlights_html = \"<div style='display: flex; flex-wrap: wrap;'>\"\n\n highlights_data['Data'] = highlights_data['Data'].apply(pd.to_numeric, errors='ignore')\n\n for index, row in highlights_data.iterrows():\n subcategory = row['SubCategory']\n data = row['Data']\n\n if subcategory in percentage_metrics:\n formatted_data = format_percentage(data)\n elif pd.notnull(data) and isinstance(data, (int, float)):\n if abs(data) >= 1_000: # Large numbers\n print(f\"{symbol}: {subcategory} is a large number. Formatting {data} with commas...\")\n formatted_data = format_large_number(data)\n else: # Other numeric data that is not a large number\n print(f\"{symbol}: {subcategory} is a numeric value. Formatting {data} with commas...\")\n formatted_data = f\"{data:,.2f}\"\n else:\n print(f'{symbol}: {subcategory} is a non-numeric value. Data: {data}')\n formatted_data = data # For non-numeric data, leave as is\n\n highlights_html += f\"\"\"<div style='flex: 0 0 calc(33.333% - 10px); margin-right: 10px; margin-bottom: 10px;'>\n <strong>{subcategory}<\/strong>: {formatted_data}\n <\/div>\"\"\"\n highlights_html += \"<\/div>\"\n new_html_content += highlights_html # App\n\n # Add the disclaimer to the end of new_html_content\n disclaimer = \"<br><br><p><strong>Disclaimer:<\/strong> The last data point has been excluded from the Prophet prediction.<\/p>\"\n new_html_content += disclaimer\n\n\n with open(html_file_path, 'r', encoding='utf-8') as f:\n html_content = f.read()\n\n # Parse the HTML content with BeautifulSoup\n soup = BeautifulSoup(html_content, 'html.parser')\n\n # Find the div that contains the Plotly plot\n plot_div = soup.find('div', {'class': 'plotly-graph-div'})\n\n if plot_div:\n # Create a BeautifulSoup object from the new_content string\n new_content_soup = BeautifulSoup(new_html_content, 'html.parser')\n\n # Insert the new content before the Plotly plot\n plot_div.insert_before(new_content_soup)\n\n # Save the modified HTML back to disk\n with open(html_file_path, 'w', encoding='utf-8') as f:\n f.write(str(soup))\n else:\n print(f\"Plotly plot not found in the HTML file {html_file_path}.\")\n\n\ndef forecast_with_multiple_metrics(df: pd.DataFrame, periods: int = 4, save_dir: str = \"plots\/\", use_all_data: bool = True, eod_price_data: str = 'eod_price_data.h5'):\n def plot_adjusted_close_from_h5(fig, symbol, hdf5_file_path, max_retries=3, sleep_duration=5):\n hdf5_file_path = data_dir + hdf5_file_path\n keys = access_hdf5_with_retries(hdf5_file_path, 'r', max_retries, sleep_duration)\n\n if f'\/{symbol}' in keys:\n with pd.HDFStore(hdf5_file_path, 'r') as store:\n stock_data = store.get(symbol)\n\n # Add actual adjusted_close to the plot\n fig.add_trace(\n go.Scatter(x=stock_data.index, y=stock_data['Adjusted_close'],\n mode='lines+markers',\n name='Actual Adjusted_close',\n legendgroup='Actual',\n line=dict(color='purple'),\n showlegend=True),\n row=1, col=1\n )\n\n prophet_df = stock_data.reset_index()[['Date', 'Adjusted_close']].rename(\n columns={'Date': 'ds', 'Adjusted_close': 'y'})\n\n best_mae = float('inf')\n best_forecast = None\n best_mode = None\n\n for mode in ['additive', 'multiplicative']:\n model = Prophet(\n seasonality_mode=mode,\n yearly_seasonality=True,\n weekly_seasonality=False,\n daily_seasonality=False)\n model.fit(prophet_df)\n future = model.make_future_dataframe(periods=365) # 1-year prediction\n forecast = model.predict(future)\n\n mae = calculate_mae(prophet_df['y'], forecast.loc[:len(prophet_df) - 1, 'yhat'])\n\n if mae < best_mae:\n best_mae = mae\n best_forecast = forecast\n best_mode = mode\n\n # Add the best forecast to the plot\n fig.add_trace(\n go.Scatter(x=best_forecast['ds'], y=best_forecast['yhat'],\n mode='lines',\n name='Adjusted_close Forecast',\n legendgroup='Forecast',\n line=dict(color='green'),\n showlegend=True),\n row=1, col=1\n )\n\n # Add yhat_upper and yhat_lower to the plot\n fig.add_trace(\n go.Scatter(x=best_forecast['ds'], y=best_forecast['yhat_upper'],\n mode='lines',\n name='Upper Forecast',\n legendgroup='Upper Forecast',\n line=dict(color='blue', dash='dash'),\n showlegend=True),\n row=1, col=1\n )\n\n fig.add_trace(\n go.Scatter(x=best_forecast['ds'], y=best_forecast['yhat_lower'],\n mode='lines',\n name='Lower Forecast',\n legendgroup='Lower Forecast',\n line=dict(color='blue', dash='dash'),\n showlegend=True),\n row=1, col=1\n )\n\n else:\n logging.warning(f\"No Adjusted_close data found for symbol: {symbol}\")\n\n pd.set_option('display.max_columns', None)\n logging.debug(df.columns.tolist())\n\n # Get the columns that are both in df.columns and reordered_columns\n common_columns = [col for col in reordered_columns if col in df.columns]\n\n # Check if common_columns is not empty\n if common_columns:\n # Reorder the DataFrame using the common columns\n df = df[common_columns + [col for col in df.columns if col not in common_columns]]\n else:\n logging.warning(\"No common columns between df and reordered_columns.\")\n\n metrics = [col for col in df.columns if df[col].dtype in ['int64', 'float64']]\n\n logging.info(metrics)\n\n subplot_titles = []\n for metric in metrics:\n definition = metric_definitions.get(metric, 'No definition available')\n wrapped_definition = wrap_text(definition, 160) # Adjust the max_length as needed\n title = f\"<b>{metric}<\/b><br><span style='font-size: smaller;'>{wrapped_definition}<\/span>\"\n subplot_titles.append(title)\n\n fig = make_subplots(rows=len(subplot_titles) + 1, cols=1, subplot_titles=['Adjusted_close'] + subplot_titles)\n\n symbol = df['Symbol'].iloc[0]\n\n plot_adjusted_close_from_h5(fig, symbol, eod_price_data)\n\n\n\n\n # Fetch the company_name\n h5_symbols = os.path.join(data_dir, \"symbols.h5\")\n company_name = get_company_name(symbol, h5_symbols)\n\n if not os.path.exists(save_dir):\n os.makedirs(save_dir)\n\n if not use_all_data:\n disclaimer = \"Disclaimer: The last data point has been excluded from the Prophet prediction. \\n \"\n disclaimer += \"There may not be enough data points for Prophet to make accurate predictions. \\n\"\n fig.add_annotation(\n dict(\n x=0,\n y=1.1,\n xref=\"paper\",\n yref=\"paper\",\n text=disclaimer,\n showarrow=False,\n font=dict(size=16)\n )\n )\n\n\n for i, metric in enumerate(metrics):\n row = i + 2\n\n plotting_df = df[['date', metric]].copy().dropna()\n prophet_df = plotting_df.copy()\n\n # Convert the specified metrics to percentages by multiplying by 100\n if metric in ['ROE', 'Earnings_Yield', 'Dividend_Yield']:\n prophet_df[metric] = prophet_df[metric] * 100 # Convert to percentage\n plotting_df[metric] = plotting_df[metric] * 100 # Convert to percentage\n\n\n if not use_all_data:\n prophet_df = prophet_df.iloc[:-1, :]\n\n prophet_df.rename(columns={'date': 'ds', metric: 'y'}, inplace=True)\n\n if prophet_df.shape[0] < 2:\n logging.warning(f\"Skipping {metric} because it has less than 2 non-NaN rows.\")\n continue\n\n best_mae = float('inf')\n best_forecast = None\n best_mode = None\n\n for mode in ['additive', 'multiplicative']:\n model = Prophet(\n seasonality_mode=mode,\n yearly_seasonality=True,\n weekly_seasonality=False,\n daily_seasonality=False)\n model.fit(prophet_df)\n future = model.make_future_dataframe(periods=periods, freq='Q')\n forecast = model.predict(future)\n\n mae = calculate_mae(prophet_df['y'], forecast.loc[:len(prophet_df) - 1, 'yhat'])\n\n if mae < best_mae:\n best_mae = mae\n best_forecast = forecast\n best_mode = mode\n\n logging.info(f\"Best seasonality mode for {metric} is {best_mode} with MAE {best_mae}\")\n\n\n hover_format = '%{x}: %{y:.2f}%' if metric in ['ROE', 'Earnings_Yield', 'Dividend_Yield'] else '%{x}: %{y:,}'\n\n fig.add_trace(\n go.Scatter(x=plotting_df['date'], y=plotting_df[metric],\n mode='lines+markers',\n name=f'Actual {metric}',\n legendgroup='Actual',\n line=dict(color='purple'),\n showlegend=(i == 0),\n hovertemplate=hover_format),\n row=row, col=1\n )\n\n # Plot the forecasted metric data\n fig.add_trace(\n go.Scatter(x=best_forecast['ds'],\n y=best_forecast['yhat'],\n mode='lines+markers',\n name=f'Forecasted {metric}',\n legendgroup='Forecast',\n line=dict(color='green'),\n showlegend=(i == 0),\n hovertemplate=hover_format),\n row=row, col=1\n )\n\n # Plot the upper forecast interval\n fig.add_trace(\n go.Scatter(x=best_forecast['ds'], y=best_forecast['yhat_upper'],\n mode='lines',\n name=f'Upper Bound {metric}',\n legendgroup='Upper Forecast',\n line=dict(color='blue', dash='dash'),\n showlegend=(i == 0),\n hovertemplate=hover_format),\n row=row, col=1\n )\n\n # Plot the lower forecast interval\n fig.add_trace(\n go.Scatter(x=best_forecast['ds'], y=best_forecast['yhat_lower'],\n mode='lines',\n name=f'Lower Bound {metric}',\n legendgroup='Lower Forecast',\n line=dict(color='blue', dash='dash'),\n showlegend=(i == 0),\n hovertemplate=hover_format),\n row=row, col=1\n )\n\n\n fig.update_layout(\n height=350 * len(metrics),\n width=1800,\n title_font_size=16 # You can change this value as needed\n )\n\n plot_file_path = os.path.join(save_dir, f\"{symbol}_quarterly.html\")\n\n fig.write_html(plot_file_path)\n logging.info(f\"Saved Prophet plot for {company_name} to {plot_file_path}\")\n\n add_content_before_plot(symbol)\n\ndef get_symbols(h5_file_path, key='US'):\n \"\"\"\n Open an HDF5 file and populate the global dictionary symbol_exchange_map\n where the symbol is the key and the exchange is the value.\n\n Parameters:\n h5_file_path (str): The path to the HDF5 file.\n key (str): The key to use when reading the HDF5 file. Default is 'US'.\n\n Returns:\n None\n \"\"\"\n\n h5_file_path = Path(h5_file_path)\n\n # Check if the file exists\n if not h5_file_path.exists():\n logging.info(f\"The file {h5_file_path} does not exist.\")\n return\n\n try:\n # Read the DataFrame from the HDF5 file\n df = pd.read_hdf(h5_file_path, key=key)\n\n # Check if 'Code' and 'Exchange' columns exist\n if 'Code' not in df.columns or 'Exchange' not in df.columns:\n logging.info(f\"The 'Code' or 'Exchange' column does not exist in the DataFrame.\")\n return\n\n # Populate the global symbol_exchange_map\n global symbol_exchange_map\n symbol_exchange_map = dict(zip(df['Code'], df['Exchange']))\n return list(symbol_exchange_map.keys())\n except Exception as e:\n logging.error(f\"An error occurred: {e}\")\n return\n\nsymbols = get_symbols(data_dir + 'symbols.h5', key='US')\n\nsymbols = ['UBER', 'LYFT', 'WE', 'IEP', 'AAPL'] + sp500 + largest_banks + companies_with_treasuries + homebuilders\nsymbols = ['GDHG', 'IEP']\nsymbols = ['GOOG']\n\n\n\nh5_files = set() # Use a set to automatically handle duplicates\n\nfor symbol in symbols:\n exchange = symbol_exchange_map.get(symbol, 'Other') # Default to 'Other' if not found\n current_h5_files = [\n f\"{data_dir}\/{exchange}_Cash_Flow_quarterly.h5\",\n f\"{data_dir}\/{exchange}_Balance_Sheet_quarterly.h5\",\n f\"{data_dir}\/{exchange}_Income_Statement_quarterly.h5\",\n f\"{data_dir}\/{exchange}_Ratios.h5\"\n ]\n h5_files.update(current_h5_files) # Update the set with the new file paths\n\n logging.info(f'Processing {symbol}')\n try:\n # Pass only the HDF5 files corresponding to the current symbol's exchange\n df = fetch_data_for_symbol_from_multiple_h5(list(current_h5_files), symbol, max_retries=3, sleep_duration=5)\n\n\n if df is None:\n logging.warning(f\"No data found for symbol {symbol}. Skipping to the next symbol.\")\n continue # Skip to the next iteration of the loop forecast_with_multiple_metrics(df, periods=4, save_dir=\"plots\/\", use_all_data=False)\n\n logging.info(df.head(20))\n forecast_with_multiple_metrics(df)\n except Exception as e:\n logging.error(f\"An unexpected error occurred while processing symbol {symbol}: {e}\")<\/code><\/pre>\n\n\n\n