2020
11.
Larson, Stefan; Mahendran, Anish; Lee, Andrew; Kummerfeld, Jonathan K; Hill, Parker; Laurenzano, Michael A; Hauswald, Johann; Tang, Lingjia; Mars, Jason
Systems and methods for automatically configuring training data for training machine learning models of a machine learning-based dialogue system including seeding training samples or curating a corpus of training data based on instances of training data identified as anomalous Miscellaneous
2020, (US Patent 10,679,150).
@misc{larson2020systems,
title = {Systems and methods for automatically configuring training data for training machine learning models of a machine learning-based dialogue system including seeding training samples or curating a corpus of training data based on instances of training data identified as anomalous},
author = {Stefan Larson and Anish Mahendran and Andrew Lee and Jonathan K Kummerfeld and Parker Hill and Michael A Laurenzano and Johann Hauswald and Lingjia Tang and Jason Mars},
url = {https://www.jasonmars.org/wp-content/uploads/2020/12/US9117447.pdf},
year = {2020},
date = {2020-06-01},
abstract = {A system and method for improving a machine learning-based dialogue system includes: sourcing a corpus of raw machine learning training data from sources of training data based on a plurality of seed training samples, wherein the corpus of raw machine learning training data comprises a plurality of distinct instances of training data; generating a vector representation for each distinct instance of training data; identifying statistical characteristics of the corpus of raw machine learning training data based on a mapping of the vector representation for each distinct instance of training data; identifying anomalous instances of the plurality of distinct instances of training data of the corpus of raw machine learning training data based on the identified statistical characteristics of the corpus; and curating the corpus of raw machine learning training data based on each of the instances of training data identified as anomalous instances.},
note = {US Patent 10,679,150},
keywords = {},
pubstate = {published},
tppubtype = {misc}
}
A system and method for improving a machine learning-based dialogue system includes: sourcing a corpus of raw machine learning training data from sources of training data based on a plurality of seed training samples, wherein the corpus of raw machine learning training data comprises a plurality of distinct instances of training data; generating a vector representation for each distinct instance of training data; identifying statistical characteristics of the corpus of raw machine learning training data based on a mapping of the vector representation for each distinct instance of training data; identifying anomalous instances of the plurality of distinct instances of training data of the corpus of raw machine learning training data based on the identified statistical characteristics of the corpus; and curating the corpus of raw machine learning training data based on each of the instances of training data identified as anomalous instances.
10.
Mars, Jason; Tang, Lingjia; Laurenzano, Michael; Hauswald, Johann; Hill, Parker
System and method for implementing an artificially intelligent virtual assistant using machine learning Miscellaneous
2020, (US Patent 10,572,801).
@misc{mars2020system,
title = {System and method for implementing an artificially intelligent virtual assistant using machine learning},
author = {Jason Mars and Lingjia Tang and Michael Laurenzano and Johann Hauswald and Parker Hill},
url = {https://www.jasonmars.org/wp-content/uploads/2020/04/US20190130244A1.pdf},
year = {2020},
date = {2020-02-01},
abstract = {Systems and methods for implementing an artificially intelligent virtual assistant includes collecting a user query; using a competency classification machine learning model to generate a competency label for the user query; using a slot identification machine learning model to segment the text of the query and label each of the slots of the query; generating a slot value for each of the slots of the query; generating a handler for each of the slot values; and using the slot values to: identify an external data source relevant to the user query, fetch user data from the external data source, and apply one or more operations to the query to generate response data; and using the response data, to generate a response to the user query.},
note = {US Patent 10,572,801},
keywords = {},
pubstate = {published},
tppubtype = {misc}
}
Systems and methods for implementing an artificially intelligent virtual assistant includes collecting a user query; using a competency classification machine learning model to generate a competency label for the user query; using a slot identification machine learning model to segment the text of the query and label each of the slots of the query; generating a slot value for each of the slots of the query; generating a handler for each of the slot values; and using the slot values to: identify an external data source relevant to the user query, fetch user data from the external data source, and apply one or more operations to the query to generate response data; and using the response data, to generate a response to the user query.
9.
Kang, Yiping; Zhang, Yunqi; Kummerfeld, Jonathan K; Hill, Parker; Hauswald, Johann; Laurenzano, Michael A; Tang, Lingjia; Mars, Jason
Systems and methods for intelligently curating machine learning training data and improving machine learning model performance Miscellaneous
2020, (US Patent 10,679,100).
@misc{kang2020systems,
title = {Systems and methods for intelligently curating machine learning training data and improving machine learning model performance},
author = {Yiping Kang and Yunqi Zhang and Jonathan K Kummerfeld and Parker Hill and Johann Hauswald and Michael A Laurenzano and Lingjia Tang and Jason Mars},
url = {https://www.jasonmars.org/wp-content/uploads/2020/12/US10679100.pdf},
year = {2020},
date = {2020-01-01},
abstract = {Systems and methods of intelligent formation and acquisition of machine learning training data for implementing an artificially intelligent dialogue system includes constructing a corpora of machine learning test corpus that comprise a plurality of historical queries and commands sampled from production logs of a deployed dialogue system; configuring training data sourcing parameters to source a corpora of raw machine learning training data from remote sources of machine learning training data; calculating efficacy metrics of the corpora of raw machine learning training data, wherein calculating the efficacy metrics includes calculating one or more of a coverage metric value and a diversity metric value of the corpora of raw machine learning training data; using the corpora of raw machine learning training data to train the at least one machine learning classifier if the calculated coverage metric value of the corpora of machine learning training data satisfies a minimum coverage metric threshold.},
note = {US Patent 10,679,100},
keywords = {},
pubstate = {published},
tppubtype = {misc}
}
Systems and methods of intelligent formation and acquisition of machine learning training data for implementing an artificially intelligent dialogue system includes constructing a corpora of machine learning test corpus that comprise a plurality of historical queries and commands sampled from production logs of a deployed dialogue system; configuring training data sourcing parameters to source a corpora of raw machine learning training data from remote sources of machine learning training data; calculating efficacy metrics of the corpora of raw machine learning training data, wherein calculating the efficacy metrics includes calculating one or more of a coverage metric value and a diversity metric value of the corpora of raw machine learning training data; using the corpora of raw machine learning training data to train the at least one machine learning classifier if the calculated coverage metric value of the corpora of machine learning training data satisfies a minimum coverage metric threshold.
8.
Mars, Jason; Tang, Lingjia; Laurenzano, Michael A; Hauswald, Johann; Hill, Parker; Kang, Yiping; Zhang, Yunqi
Systems and methods for intelligently configuring and deploying a machine learning-based dialogue system Miscellaneous
2020, (US Patent 10,769,384).
@misc{mars2020systems,
title = {Systems and methods for intelligently configuring and deploying a machine learning-based dialogue system},
author = {Jason Mars and Lingjia Tang and Michael A Laurenzano and Johann Hauswald and Parker Hill and Yiping Kang and Yunqi Zhang},
url = {https://www.jasonmars.org/wp-content/uploads/2020/12/US10769384.pdf},
year = {2020},
date = {2020-01-01},
abstract = {A system and method for intelligently configuring a machine learning-based dialogue system includes a conversational deficiency assessment of a target dialog system, wherein implementing the conversational deficiency assessment includes: (i) identifying distinct corpora of mishandled utterances based on an assessment of the distinct corpora of dialogue data; (ii) identifying candidate corpus of mishandled utterances from the distinct corpora of mishandled utterances as suitable candidates for building new dialogue competencies for the target dialogue system if candidate metrics of the candidate corpus of mishandled utterances satisfy a candidate threshold; building the new dialogue competencies for the target dialogue system for each of the candidate corpus of mishandled utterances having candidate metrics that satisfy the candidate threshold; and configuring a dialogue system control structure for the target dialogue system based on the new dialogue competencies, wherein the dialogue system control structure governs an operation of an automated dialogue agent.},
note = {US Patent 10,769,384},
keywords = {},
pubstate = {published},
tppubtype = {misc}
}
A system and method for intelligently configuring a machine learning-based dialogue system includes a conversational deficiency assessment of a target dialog system, wherein implementing the conversational deficiency assessment includes: (i) identifying distinct corpora of mishandled utterances based on an assessment of the distinct corpora of dialogue data; (ii) identifying candidate corpus of mishandled utterances from the distinct corpora of mishandled utterances as suitable candidates for building new dialogue competencies for the target dialogue system if candidate metrics of the candidate corpus of mishandled utterances satisfy a candidate threshold; building the new dialogue competencies for the target dialogue system for each of the candidate corpus of mishandled utterances having candidate metrics that satisfy the candidate threshold; and configuring a dialogue system control structure for the target dialogue system based on the new dialogue competencies, wherein the dialogue system control structure governs an operation of an automated dialogue agent.
7.
Peper, Joseph; Hill, Parker; Leach, Kevin; Stapleton, Sean; Kummerfeld, Jonathan K; Hauswald, Johann; Laurenzano, Michael; Tang, Lingjia; Mars, Jason
Systems and methods for machine learning-based multi-intent segmentation and classification Miscellaneous
2020, (US Patent 10,824,818).
@misc{peper2020systems,
title = {Systems and methods for machine learning-based multi-intent segmentation and classification},
author = {Joseph Peper and Parker Hill and Kevin Leach and Sean Stapleton and Jonathan K Kummerfeld and Johann Hauswald and Michael Laurenzano and Lingjia Tang and Jason Mars},
url = {https://www.jasonmars.org/wp-content/uploads/2020/12/US10824818.pdf},
year = {2020},
date = {2020-01-01},
abstract = {Systems and methods for synthesizing training data for multi-intent utterance segmentation include identifying a first corpus of utterances comprising a plurality of distinct single-intent in-domain utterances; identifying a second corpus of utterances comprising a plurality of distinct single-intent out-of-domain utterances; identifying a third corpus comprising a plurality of distinct conjunction terms; forming a multi-intent training corpus comprising synthetic multi-intent utterances, wherein forming each distinct multi-intent utterance includes: selecting a first distinct in-domain utterance from the first corpus of utterances; probabilistically selecting one of a first out-of-domain utterance from the second corpus and a second in-domain utterance from the first corpus; probabilistically selecting or not selecting a distinct conjunction term from the third corpus; and forming a synthetic multi-intent utterance including appending the first in-domain utterance with one of the first out-of-domain utterance from the second corpus of utterances and the second in-domain utterance from the first corpus of utterances.},
note = {US Patent 10,824,818},
keywords = {},
pubstate = {published},
tppubtype = {misc}
}
Systems and methods for synthesizing training data for multi-intent utterance segmentation include identifying a first corpus of utterances comprising a plurality of distinct single-intent in-domain utterances; identifying a second corpus of utterances comprising a plurality of distinct single-intent out-of-domain utterances; identifying a third corpus comprising a plurality of distinct conjunction terms; forming a multi-intent training corpus comprising synthetic multi-intent utterances, wherein forming each distinct multi-intent utterance includes: selecting a first distinct in-domain utterance from the first corpus of utterances; probabilistically selecting one of a first out-of-domain utterance from the second corpus and a second in-domain utterance from the first corpus; probabilistically selecting or not selecting a distinct conjunction term from the third corpus; and forming a synthetic multi-intent utterance including appending the first in-domain utterance with one of the first out-of-domain utterance from the second corpus of utterances and the second in-domain utterance from the first corpus of utterances.
6.
Lee, Andrew; Larson, Stefan; Clarke, Christopher; Leach, Kevin; Kummerfeld, Jonathan K; Hill, Parker; Hauswald, Johann; Laurenzano, Michael A; Tang, Lingjia; Mars, Jason; others,
Systems and methods for constructing an artificially diverse corpus of training data samples for training a contextually-biased model for a machine learning-based dialogue system Miscellaneous
2020, (US Patent 10,796,104).
@misc{lee2020systems,
title = {Systems and methods for constructing an artificially diverse corpus of training data samples for training a contextually-biased model for a machine learning-based dialogue system},
author = {Andrew Lee and Stefan Larson and Christopher Clarke and Kevin Leach and Jonathan K Kummerfeld and Parker Hill and Johann Hauswald and Michael A Laurenzano and Lingjia Tang and Jason Mars and others},
url = {https://www.jasonmars.org/wp-content/uploads/2020/12/US10796104.pdf},
year = {2020},
date = {2020-01-01},
abstract = {Systems and methods for constructing an artificially diverse corpus of training data includes evaluating a corpus of utterance-based training data samples, identifying a slot replacement candidate; deriving distinct skeleton utterances that include the slot replacement candidate, wherein deriving the distinct skeleton utterances includes replacing slots of each of the plurality of distinct utterance training samples with one of a special token and proper slot classification labels; selecting a subset of the distinct skeleton utterances; converting each of the distinct skeleton utterances of the subset back to distinct utterance training samples while still maintaining the special token at a position of the slot replacement candidate; altering a percentage of the distinct utterance training samples with a distinct randomly-generated slot token value at the position of the slot replacement candidate; and constructing the artificially diverse corpus of training samples based on a collection of the percentage of the distinct utterance training samples.},
note = {US Patent 10,796,104},
keywords = {},
pubstate = {published},
tppubtype = {misc}
}
Systems and methods for constructing an artificially diverse corpus of training data includes evaluating a corpus of utterance-based training data samples, identifying a slot replacement candidate; deriving distinct skeleton utterances that include the slot replacement candidate, wherein deriving the distinct skeleton utterances includes replacing slots of each of the plurality of distinct utterance training samples with one of a special token and proper slot classification labels; selecting a subset of the distinct skeleton utterances; converting each of the distinct skeleton utterances of the subset back to distinct utterance training samples while still maintaining the special token at a position of the slot replacement candidate; altering a percentage of the distinct utterance training samples with a distinct randomly-generated slot token value at the position of the slot replacement candidate; and constructing the artificially diverse corpus of training samples based on a collection of the percentage of the distinct utterance training samples.
2019
5.
Kang, Yiping; Zhang, Yunqi; Kummerfeld, Jonathan K; Hill, Parker; Hauswald, Johann; Laurenzano, Michael A; Tang, Lingjia; Mars, Jason
Systems and methods for intelligently curating machine learning training data and improving machine learning model performance Miscellaneous
2019, (US Patent 10,303,978).
@misc{kang2019systems,
title = {Systems and methods for intelligently curating machine learning training data and improving machine learning model performance},
author = {Yiping Kang and Yunqi Zhang and Jonathan K Kummerfeld and Parker Hill and Johann Hauswald and Michael A Laurenzano and Lingjia Tang and Jason Mars},
url = {https://www.jasonmars.org/wp-content/uploads/2020/04/US20190294925A1.pdf},
year = {2019},
date = {2019-05-01},
abstract = {Systems and methods of intelligent formation and acquisition of machine learning training data for implementing an artificially intelligent dialogue system includes constructing a corpora of machine learning test corpus that comprise a plurality of historical queries and commands sampled from production logs of a deployed dialogue system; configuring training data sourcing parameters to source a corpora of raw machine learning training data from remote sources of machine learning training data; calculating efficacy metrics of the corpora of raw machine learning training data, wherein calculating the efficacy metrics includes calculating one or more of a coverage metric value and a diversity metric value of the corpora of raw machine learning training data; using the corpora of raw machine learning training data to train the at least one machine learning classifier if the calculated coverage metric value of the corpora of machine learning training data satisfies a minimum coverage metric threshold.},
note = {US Patent 10,303,978},
keywords = {},
pubstate = {published},
tppubtype = {misc}
}
Systems and methods of intelligent formation and acquisition of machine learning training data for implementing an artificially intelligent dialogue system includes constructing a corpora of machine learning test corpus that comprise a plurality of historical queries and commands sampled from production logs of a deployed dialogue system; configuring training data sourcing parameters to source a corpora of raw machine learning training data from remote sources of machine learning training data; calculating efficacy metrics of the corpora of raw machine learning training data, wherein calculating the efficacy metrics includes calculating one or more of a coverage metric value and a diversity metric value of the corpora of raw machine learning training data; using the corpora of raw machine learning training data to train the at least one machine learning classifier if the calculated coverage metric value of the corpora of machine learning training data satisfies a minimum coverage metric threshold.
4.
Tang, Lingjia; Mars, Jason; Hundt, Robert
System and methods for sharing memory subsystem resources among datacenter applications Miscellaneous
2019, (US Patent 10,313,265).
@misc{tang2019system,
title = {System and methods for sharing memory subsystem resources among datacenter applications},
author = {Lingjia Tang and Jason Mars and Robert Hundt},
url = {https://www.jasonmars.org/wp-content/uploads/2020/04/US9401869.pdf},
year = {2019},
date = {2019-01-01},
abstract = {Systems and methods for mapping applications onto system resource of a computing platform are discussed. The computing platform may receive, using control circuitry, a request to run a plurality of applications on a computing platform having a plurality of system resources. The computing platform may determine a plurality of mapping configurations for the plurality of applications onto the plurality of system resources. The computing platform may execute the plurality of applications with each of the plurality of mapping configurations. The computing platform may determine at least one performance metric based on the executed plurality of applications for each of the plurality of mapping configurations. The computing platform may select a selected mapping configuration among the plurality of mapping configurations based on at least one determined performance metric.},
note = {US Patent 10,313,265},
keywords = {},
pubstate = {published},
tppubtype = {misc}
}
Systems and methods for mapping applications onto system resource of a computing platform are discussed. The computing platform may receive, using control circuitry, a request to run a plurality of applications on a computing platform having a plurality of system resources. The computing platform may determine a plurality of mapping configurations for the plurality of applications onto the plurality of system resources. The computing platform may execute the plurality of applications with each of the plurality of mapping configurations. The computing platform may determine at least one performance metric based on the executed plurality of applications for each of the plurality of mapping configurations. The computing platform may select a selected mapping configuration among the plurality of mapping configurations based on at least one determined performance metric.
2017
3.
Hundt, Robert; Tang, Lingjia; Mars, Jason
Allocation of tasks in large scale computing systems Miscellaneous
2017, (US Patent 9,563,532).
@misc{hundt2017allocation,
title = {Allocation of tasks in large scale computing systems},
author = {Robert Hundt and Lingjia Tang and Jason Mars},
url = {https://www.jasonmars.org/wp-content/uploads/2020/04/pat9563532.pdf},
year = {2017},
date = {2017-02-01},
abstract = {Aspects of the invention may be used to allocate tasks among computing machines in large scale computing systems. In one aspect, the method includes executing a first task in the plurality of tasks on a first computing machine and determining a performance degradation threshold for the first task. The method further includes calculating a predicted performance degradation of the first task when a second task is executed on the first computing machine, wherein the predicted performance degradation is determined by comparing a performance interference score of the second task with a performance sensitivity curve of the first task. The method further includes executing the second task on the first computing machine when the predicted performance degradation of the first task is below the performance degradation threshold.},
note = {US Patent 9,563,532},
keywords = {},
pubstate = {published},
tppubtype = {misc}
}
Aspects of the invention may be used to allocate tasks among computing machines in large scale computing systems. In one aspect, the method includes executing a first task in the plurality of tasks on a first computing machine and determining a performance degradation threshold for the first task. The method further includes calculating a predicted performance degradation of the first task when a second task is executed on the first computing machine, wherein the predicted performance degradation is determined by comparing a performance interference score of the second task with a performance sensitivity curve of the first task. The method further includes executing the second task on the first computing machine when the predicted performance degradation of the first task is below the performance degradation threshold.
2013
2.
Mars, Jason; Hundt, Robert
Scenario based optimization Miscellaneous
2013, (US Patent 8,578,355).
@misc{mars2013scenario,
title = {Scenario based optimization},
author = {Jason Mars and Robert Hundt},
url = {https://www.jasonmars.org/wp-content/uploads/2020/05/US8578355.pdf},
year = {2013},
date = {2013-01-01},
abstract = {Techniques and systems for scenario based optimization can include generating multiple different versions of a program segment based on different respective execution scenarios associated with an execution of a program, the program operable to use the program segment versions. In another aspect, techniques and systems can include executing a program executable associated with multiple different versions of a program segment, analyzing the execution for an indication of at least one of the execution scenarios to select one of the program segment versions based on the indication, and causing the execution to use the selected program segment version during at least a portion of the execution.},
note = {US Patent 8,578,355},
keywords = {},
pubstate = {published},
tppubtype = {misc}
}
Techniques and systems for scenario based optimization can include generating multiple different versions of a program segment based on different respective execution scenarios associated with an execution of a program, the program operable to use the program segment versions. In another aspect, techniques and systems can include executing a program executable associated with multiple different versions of a program segment, analyzing the execution for an indication of at least one of the execution scenarios to select one of the program segment versions based on the indication, and causing the execution to use the selected program segment version during at least a portion of the execution.
0000
1.
Mars, Jason; Tang, Lingjia; Laurenzano, Michael A; Hauswald, Johann; Hill, Parker; Kang, Yiping; Zhang, Yunqi
Systems and methods for intelligently configuring and deploying a machine learning-based dialogue system Miscellaneous
0000, (US Patent 10,740,371).
@misc{mars2020systemsb,
title = {Systems and methods for intelligently configuring and deploying a machine learning-based dialogue system},
author = {Jason Mars and Lingjia Tang and Michael A Laurenzano and Johann Hauswald and Parker Hill and Yiping Kang and Yunqi Zhang},
note = {US Patent 10,740,371},
keywords = {},
pubstate = {published},
tppubtype = {misc}
}