| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-03-23 | |||
| 17:38:11 | sean-k-mooney | hrw: you are using a non zuulv3 native job yes? | |
| 17:38:34 | sean-k-mooney | hrw: im guessing the logs you are looking at are double compressed | |
| 17:38:52 | hrw | sean-k-mooney: running tempest locally | |
| 17:39:08 | sean-k-mooney | if that is the issue then you can fix it by doing "curl <log url> | zcat | <program of choice or file>" | |
| 17:39:17 | sean-k-mooney | oh so its not that issue | |
| 17:39:36 | gibi | dansmith: responded. I think the current code is OK, that loop was never designed to try multiple hosts for a single instance | |
| 17:40:12 | sean-k-mooney | gibi: but should it contiue to the next iteration and try the other instnaces? | |
| 17:40:40 | gibi | sean-k-mooney: honestly I would not try that right now. Separately we can think about such enhancement | |
| 17:40:42 | hrw | sean-k-mooney: I start to think that at the end I will write a python script which with load that 10-20MB json file into memory and parse to provide some sane report | |
| 17:41:07 | gibi | sean-k-mooney: this is now simple and it cleans up properly as far as I see. | |
| 17:41:13 | sean-k-mooney | gibi: ok dansmith raised that question in https://review.opendev.org/#/c/631244/68/nova/conductor/manager.py@1621 | |
| 17:41:44 | gibi | and I responded there | |
| 17:42:00 | sean-k-mooney | hrw: there is a tool upstream that renders a nice html output | |
| 17:42:03 | dansmith | otp, bbl | |
| 17:42:18 | sean-k-mooney | gibi: cool havent reloaded | |
| 17:42:27 | sean-k-mooney | im going to move on to the next patch in the chain | |
| 17:42:46 | gibi | sean-k-mooney: thanks. | |
| 17:42:51 | gibi | I'm going offline :) | |
| 17:42:53 | gibi | o/ | |
| 17:44:50 | hrw | sean-k-mooney: it would be too good. | |
| 18:01:42 | hrw | sean-k-mooney: thanks | |
| 19:03:42 | artom | dansmith, got the energy to look at https://review.opendev.org/#/c/714228/ again? | |
| 19:04:53 | openstackgerrit | Merged openstack/nova master: Make serialize_args handle exception messages safely https://review.opendev.org/712697 | |
| 19:08:34 | artom | stephenfin ^^ same, but you're probably done for the day by now | |
| 19:24:35 | openstackgerrit | Merged openstack/nova master: Store instance action event exc_val fault details https://review.opendev.org/694428 | |
| 19:24:44 | openstackgerrit | Merged openstack/nova master: Enable unshelve with qos ports https://review.opendev.org/705475 | |
| 19:38:09 | openstackgerrit | Merged openstack/nova master: Merge qos related renos for Ussuri https://review.opendev.org/706766 | |
| 20:12:57 | spatel | sean-k-mooney: is rcp_thread_pool_size has been replaced with executor_thread_pool_size ? | |
| 20:13:04 | spatel | https://docs.openstack.org/nova/queens/configuration/config.html#executor_thread_pool_size | |
| 20:14:18 | sean-k-mooney | https://docs.openstack.org/nova/queens/configuration/config.html#DEFAULT.executor_thread_pool_size | |
| 20:14:31 | sean-k-mooney | according to the docs yes | |
| 20:15:39 | spatel | cool, i will use executor_thread_pool_size :) | |
| 20:32:01 | spatel | sean-k-mooney: what do you suggest about heal_instance_info_cache_interval value, its ok to keep it little high (default: 60) | |
| 20:33:01 | sean-k-mooney | in theory it should not be needed at all. it is intended to ensure that nova and neutron does not get out of sync if you modify attributes of the neutron port out of band of nova | |
| 20:33:21 | sean-k-mooney | so yes it can be set longer that said it should not be disabled | |
| 20:33:39 | sean-k-mooney | mainly because there is not testing for running without the healing. | |
| 20:34:25 | spatel | i am trying to reduce load on neutron-servers so trying to tune some value | |
| 20:35:18 | spatel | as VMs grows it put more pressure on neutron | |
| 20:38:44 | sean-k-mooney | you should be able to set it to 300 safely enough | |
| 20:41:10 | spatel | cool | |
| 21:15:00 | sean-k-mooney | dansmith: gibi Sundar just finished reviewing the cyborg series. i noticed a few nits all of which could be adressed in a follow up patch but one thing is definetly missing. i think we all agree that cyborg integration qualifies for a release note but i dont see one in the series | |
| 21:16:16 | sean-k-mooney | we could add it in the followup patch when adressing some fo the nits but strictly speaking i belive it should be added here https://review.opendev.org/#/c/673735/46 and then update by each of the follow on patches to reflect the different capablities | |
| 21:16:44 | sean-k-mooney | that said we can just add it in https://review.opendev.org/#/c/674726/9 which is the last functional patch | |
| 21:21:11 | Sundar | sean-k-mooney: Ack. I'll take a look | |
| 21:21:24 | Sundar | Thanks for the reviews. | |
| 21:21:44 | Sundar | Any luck with testing the rescheduling? | |
| 21:33:17 | sean-k-mooney | i have not had a chance to test it but i might get to do it tomororow. i will be redeploying the latest version tomorow in anycase | |
| 21:33:43 | sean-k-mooney | i was condiering locally modifying the spawn to fail on one node to then assert the correct behaviro for reschdule | |
| 21:42:06 | sean-k-mooney | artom: whitebox is breaking thing it should not be able to break... | |
| 21:42:08 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/bc5fd77019b74363ab3b06478fd386de | |
| 21:42:51 | sean-k-mooney | artom: its causing tempest-tox-plugin-sanity-check to fail | |
| 21:43:01 | sean-k-mooney | . | |
| 21:43:04 | artom | Whoa | |
| 21:43:23 | sean-k-mooney | i have no idea why its checking whitebox either | |
| 21:43:34 | sean-k-mooney | but we should proably fix that | |
| 21:44:07 | sean-k-mooney | its becasue we merged https://opendev.org/x/whitebox-tempest-plugin/commit/c1285626fee890a00f247ff0d0bde1249b90a59f | |
| 21:44:17 | artom | sean-k-mooney, right, I get the error | |
| 21:44:28 | artom | I just have no idea how we ended up in that job | |
| 21:44:40 | artom | TO my knowledge we didn't "register" anywhere as a tempest plugin | |
| 21:44:42 | sean-k-mooney | did your governance patch merge | |
| 21:44:49 | artom | No | |
| 21:45:01 | sean-k-mooney | then ya im not sure either | |
| 21:45:16 | sean-k-mooney | i guess we need to set a default for the config | |
| 21:45:27 | artom | I was gonna say - that's probably the easiest fix | |
| 21:45:27 | sean-k-mooney | so 0? | |
| 21:45:29 | artom | Yeah | |
| 21:45:38 | sean-k-mooney | shoudl we also add that job to whitebox | |
| 21:45:49 | artom | Let's fix it for now | |
| 21:46:03 | sean-k-mooney | ok want me to do it now or will i leave it to you | |
| 21:46:17 | artom | sean-k-mooney, we need each other to review it :) | |
| 21:46:46 | sean-k-mooney | yes but in this case i think we can self approve | |
| 21:46:56 | sean-k-mooney | once the other revies its also trival | |
| 21:52:05 | artom | sean-k-mooney, https://review.opendev.org/#/c/714539/ | |
| 21:52:20 | sean-k-mooney | artom: hehe you bet me too it | |
| 21:53:45 | artom | sean-k-mooney, ah, I think that tempest job dynamically gets a list of projects that are tempest plugins from gerrit: https://opendev.org/openstack/tempest/src/branch/master/tools/generate-tempest-plugins-list.py | |
| 21:54:20 | sean-k-mooney | ya so i guess its trying to make sure that changes to tempest dont break plugins | |
| 21:55:06 | artom | Risky as hell though | |
| 21:55:24 | artom | Though they can blacklist projects too | |
| 21:55:29 | artom | So maybe not to much | |
| 21:55:31 | artom | *so much | |
| 21:55:58 | artom | I'll email the ML | |
| 21:56:28 | sean-k-mooney | ya. well i think that reinforces the idea we proably should run that job on the mailing list but on the plus side they should be warned if they would breake whitebox atleast to some degree | |
| 22:01:19 | artom | sean-k-mooney, not sure how running that job would work, it depends on scripts in the tempest rpo | |
| 22:01:21 | artom | *repo | |
| 22:01:50 | artom | Anyways, we can continue this later, I need to make supper | |
| 22:02:19 | sean-k-mooney | ya im also done for the day. but runing that job would use the script form the tempest repo | |
| 22:02:41 | sean-k-mooney | it would tell use if we were making a change that woudl cause it to fail before we merge it | |
| 22:02:58 | artom | OK, so should work then | |
| 22:03:04 | artom | But yeah, tomorrow | |
| 22:03:05 | artom | o/ | |
| 22:03:10 | sean-k-mooney | o/ | |
| 23:42:52 | openstackgerrit | melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478 | |
| 23:42:54 | openstack | bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929 | |
| #openstack-nova - 2020-03-24 | |||
| 00:28:48 | openstackgerrit | Merged openstack/nova master: Add service version check for live migrate with qos https://review.opendev.org/713679 | |
| 00:29:40 | openstackgerrit | Merged openstack/nova master: Add service version check for evacuate with qos https://review.opendev.org/713680 | |
| 00:29:46 | openstackgerrit | Merged openstack/nova stable/queens: Stop setting glance_api_version in cinder.conf in nova-live-migration https://review.opendev.org/713843 | |
| 00:29:52 | openstackgerrit | Merged openstack/nova stable/queens: Replace ansible --sudo with --become in live_migration/hooks scripts https://review.opendev.org/713844 | |
| 00:29:59 | openstackgerrit | Merged openstack/nova stable/queens: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713845 | |
| 00:49:07 | openstackgerrit | melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478 | |
| 00:49:10 | openstack | bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929 | |