| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-04 | |||
| 13:56:31 | dansmith | artom: yeah that was before the hugepage work right? | |
| 13:56:43 | artom | dansmith, I believe that was after | |
| 13:56:57 | sean-k-mooney | this was after https://46b5dc7879590352c3ef-b484a2b89d5d2c358f068133dfb2fa14.ssl.cf5.rackcdn.com/679805/2/experimental/nova-nfv-multi-numa-multinode/b040d28/testr_results.html.gz | |
| 13:57:01 | artom | dansmith, 13:12 my time, I was talking to danpb that morning about hugepages | |
| 13:58:05 | sean-k-mooney | this is from the run where i forced the vcpu_pin_set and cpu_shared_set to be reversed on each host | |
| 13:58:07 | sean-k-mooney | https://46b5dc7879590352c3ef-b484a2b89d5d2c358f068133dfb2fa14.ssl.cf5.rackcdn.com/679805/2/experimental/nova-nfv-multi-numa-multinode/b040d28/compute/logs/etc/nova/nova_conf.txt.gz | |
| 13:59:12 | luyao | dansmith, alex_xu : I found the whitelist. Thanks! | |
| 13:59:31 | alex_xu | luyao: really, I'm not...still looking | |
| 14:00:15 | alex_xu | ah, I found it | |
| 14:00:23 | alex_xu | luyao: you win | |
| 14:00:23 | luyao | alex_xu: it's in nova/tests/unit/db/test_migrations.py, def filter_metadata_diff | |
| 14:00:34 | alex_xu | dansmith: thanks again | |
| 14:02:18 | yonglihe | alex_xu: Good evening. | |
| 14:02:29 | alex_xu | yonglihe: good evening too :) | |
| 14:09:26 | dansmith | artom: sean-k-mooney: What're ya'll doing to test the rollback case IRL? | |
| 14:10:00 | artom | dansmith, IRL? nothing. Trying to understand why the func test wasn't picking it up | |
| 14:10:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove Rocky compute compat checks for live migration with port bindings https://review.opendev.org/680005 | |
| 14:10:23 | dansmith | artom: okay, I would expect you'd set up something to fail the migration to make sure the rollback actually happens properly | |
| 14:10:27 | sean-k-mooney | e.g. a failure to migrate, i have inetvertenly tested it by forgettting to exchanged ssh keys and haveing the migrate fail and rollback | |
| 14:10:39 | sean-k-mooney | that is the only testing of it i did | |
| 14:10:43 | dansmith | sean-k-mooney: is it getting far enough that it actually calls the new method? | |
| 14:10:54 | sean-k-mooney | yes | |
| 14:10:58 | dansmith | okay | |
| 14:11:03 | sean-k-mooney | i have not done that on the latest version | |
| 14:11:08 | sean-k-mooney | want me to do that | |
| 14:11:23 | sean-k-mooney | i just need to drop the ssh key form autherised keys | |
| 14:11:30 | dansmith | sean-k-mooney: can you also do something else for us? pin the compute version on one machine to 5.2 and migrate both directions to make sure it still works | |
| 14:11:40 | dansmith | because we really *really* don't want to break migrations for upgrades | |
| 14:11:47 | artom | dansmith, I suppose I can setup stein/master and try it | |
| 14:11:53 | sean-k-mooney | sure. i can do that in a ci test too | |
| 14:12:06 | artom | dansmith, oh, just pinning. yeah :) | |
| 14:12:06 | dansmith | if we find out late that we did and need to tweak rpc stuff to make it work, life is gonna suck | |
| 14:12:08 | sean-k-mooney | but ill test both | |
| 14:12:15 | dansmith | sean-k-mooney: okay thanks | |
| 14:12:33 | dansmith | I see the logs show it actually doing the new stuff so that's cool | |
| 14:12:33 | sean-k-mooney | dansmith: i can also give you acess once i have it set up if you want to poke around | |
| 14:12:43 | dansmith | sean-k-mooney: that's okay, I' | |
| 14:12:44 | artom | (Wouldn't func be sufficient - like, I get wanting to see real libvirt with the updated XML, and thank god because it caught bugs) | |
| 14:12:50 | dansmith | sean-k-mooney: I'll trust some pastebin'd logs | |
| 14:13:01 | artom | (But rollback is all compute) | |
| 14:13:02 | dansmith | artom: func is great, but it's not the real thing | |
| 14:13:26 | dansmith | artom: trust me, you don't want to be on a sev1 where you realize you're going to need to add an RPC .z version to fix this | |
| 14:13:34 | dansmith | we did that in icehouse for a customer and it was not cool | |
| 14:13:48 | artom | Fully agree there | |
| 14:14:57 | artom | sean-k-mooney, how feasible would it be to modify the grenade job to run with a NUMA flavor? | |
| 14:15:04 | artom | On FN hardwa | |
| 14:15:06 | artom | *hardware | |
| 14:15:33 | sean-k-mooney | so i am less familar with legacy jobs but it could be done | |
| 14:15:54 | sean-k-mooney | the nodeset is not hard but forcing it to use nested virt is the tricky bit | |
| 14:16:05 | sean-k-mooney | devstack gate is non tivial | |
| 14:16:14 | sean-k-mooney | adding grendate on top is harder | |
| 14:16:22 | sean-k-mooney | if we need to do that we could however | |
| 14:16:43 | artom | sean-k-mooney, well, dansmith said he's satisfied with pastebin, so maybe more long term | |
| 14:16:44 | sean-k-mooney | i would like to port greadne to a non legacy job at some point | |
| 14:17:23 | sean-k-mooney | well well i have not updated https://review.opendev.org/#/c/679805/ to pick up the other changes i did this morning | |
| 14:17:41 | sean-k-mooney | i can updated it and add the different compute versions in that | |
| 14:17:55 | sean-k-mooney | ill try and test it locally first however | |
| 14:19:44 | mriedem | speaking of converting a job to zuulv3, https://review.opendev.org/#/c/674345/ converts the nova-lvm job to zuulv3 and has a +2 | |
| 14:19:46 | sean-k-mooney | dansmith: do you know how to pin the rpc version of the top of your head. ill look it up in teh config ref if not | |
| 14:20:00 | mriedem | sean-k-mooney: [upgrade_levels]/compute=stein | |
| 14:20:05 | dansmith | yeah that | |
| 14:20:10 | sean-k-mooney | thanks | |
| 14:20:24 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#upgrade_levels.compute | |
| 14:21:24 | mriedem | once that nova-lvm change is merged the only legacy jobs we have are the live migration and grenade multinode ones, which i've probably put off b/c of the external script they run, but nova-next was converted and runs an external script so i can probably do similar | |
| 14:21:24 | sean-k-mooney | does it matter if i do this on the compute or the contoler? e.g. we want the conductor ect to not be pinned right | |
| 14:21:31 | sean-k-mooney | just the source node? | |
| 14:21:38 | mriedem | sean-k-mooney: if you pin in nova-cpu.conf i don't think it matters | |
| 14:21:44 | mriedem | conductor is using a different config | |
| 14:22:14 | sean-k-mooney | ya just want to make sure i dont mess up my local.conf and have to redeploy | |
| 14:22:27 | sean-k-mooney | ill pin it on the dedicated compute node | |
| 14:22:39 | sean-k-mooney | nova-cpu.conf would also work | |
| 14:23:08 | sean-k-mooney | yes you did i was agreeing with you | |
| 14:24:08 | mriedem | i can tinker later this afternoon with converting that nova-grenade-multinode job to zuulv3 | |
| 14:24:13 | mriedem | those have been on my todo list for awhile | |
| 14:25:42 | sean-k-mooney | i can also try if you run into issue but i have never actully run greade and i dont know if we have the ansibel playbooks to run it so im not sure how easy that will be | |
| 14:26:17 | sean-k-mooney | is greade hard to run or is basically the same as devstack | |
| 14:32:03 | mriedem | it's just a set of bash scripts that orchestrates setting up n-1 devstack, creating some resources, shutting down and then running N version devstack and making sure those resources are still around | |
| 14:32:14 | mriedem | i've never run it locally manually, only in CI | |
| 14:32:33 | gibi | mriedem: looking at the MigrationTask. When the dest is selected by the scheduler I can check right away if the dest has a new enough compute service version. But if not, then what? simulate a re-schedule locally in the conductor? https://github.com/openstack/nova/blob/cbaea3bd690c683e83a063acfa69919668eaa123/nova/conductor/tasks/migrate.py#L248 | |
| 14:33:10 | mriedem | gibi: you'd have to iterate alternates | |
| 14:33:13 | sean-k-mooney | mriedem: looking at the zuul config they dont have a non legacy way to run it | |
| 14:33:19 | sean-k-mooney | https://opendev.org/openstack/grenade/src/branch/master/.zuul.yaml | |
| 14:33:21 | mriedem | it's times like this that i've thought about having a compute service version filter in the scheduler | |
| 14:33:32 | mriedem | so we can say internally "only give me computes with service version >= x for this feature" | |
| 14:34:12 | sean-k-mooney | mriedem: well if that feature was a compute capablity tratis we could use placemtn for that | |
| 14:34:14 | gibi | mriedem: Yeah that was what I mean by simualting a re-schedule locally , as I have to take care of num_retries and the allocations too | |
| 14:34:16 | mriedem | sean-k-mooney: https://review.opendev.org/#/c/548936/ | |
| 14:34:20 | sean-k-mooney | like your multi attach stuff | |
| 14:34:39 | openstackgerrit | Merged openstack/nova master: Make _get_cpu_feature_traits() always return a dict https://review.opendev.org/679568 | |
| 14:34:52 | sean-k-mooney | mriedem: ya i saw that mentioned on the infra irc yesterday | |
| 14:34:52 | openstackgerrit | Merged openstack/nova master: libvirt: Make scheduler filters customizable https://review.opendev.org/679745 | |
| 14:35:00 | sean-k-mooney | i was not sure how far along it was | |
| 14:35:00 | mriedem | gibi: num_retries, you mean max_attempts? | |
| 14:35:10 | mriedem | gibi: if so, that's built into the number of alternates you get back from the scheduler | |
| 14:35:38 | gibi | mriedem: ohh, good point. If I run out of alternates then raise MaxRetriesExceeded | |
| 14:36:17 | mriedem | gibi: yeah, and write it in a separate method so we can drop that in U or V | |
| 14:36:23 | gibi | mriedem: sure | |
| 14:37:10 | sean-k-mooney | mriedem: ok it looks like that is passing. i wonder is there a reason it is not merged yet. i should go ask them | |
| 14:37:27 | gibi | mriedem: also I think I have to check the version of the source compute as well becuase request spec travels like cond -> dest -> source -> dest so if the source is old then the request_spec will not reach the dest at the end | |
| 14:37:32 | mriedem | sean-k-mooney: the scheduler will already filter dest computes that have the required nested inventory for qos bw providers, but we do'nt know if that dest compute is stein or train to support cold migratoin with those things, and i don't think we want to get into a habit of adding a compute capability for every "supports_fancy_unicorn" feature just for rolling upgrade support | |