| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-07 | |||
| 19:33:30 | artom | And it was/is missing basic microversion support | |
| 19:33:38 | sean-k-mooney | artom: the intent is we woudl jsut update the sdk for the api changes and then have a thin wrapper in osc | |
| 19:33:55 | sean-k-mooney | but yes its moving slowly to the sdk | |
| 19:34:30 | artom | mriedem, I agree with the sentiment :) It often comes up downstream - clients or whoever will say "I can't do this with osc [which is the recommended client for interaction with our OSP product]" and we'll come back with "use novaclient instead" | |
| 19:35:37 | sean-k-mooney | artom: we only do that if there is no other choice | |
| 19:35:39 | sean-k-mooney | but yes | |
| 19:35:47 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove the TODO about using OSC for BFV in test_evacuate.sh https://review.opendev.org/693406 | |
| 19:36:01 | efried | hahaha | |
| 19:36:04 | sean-k-mooney | although we really should file a bug downstream to fix the osc gap everytime we do that | |
| 19:37:04 | efried | wasn't there a spec to allow you to flip the 'remove on instance deletion' flag after the instance was created? | |
| 19:37:19 | mriedem | yes | |
| 19:37:30 | sean-k-mooney | efried: i think they wanted to set it as part of the attachment or somthing | |
| 19:37:32 | mriedem | that would be a pretty crappy workaround here | |
| 19:37:43 | efried | https://review.opendev.org/#/c/580336/ | |
| 19:37:55 | artom | *snerk* | |
| 19:37:58 | efried | mriedem: but eventually we'll want osc support for setting that option initially, right? | |
| 19:38:25 | mriedem | i just loudly audibly sighed | |
| 19:38:27 | efried | so like, instead of removing the TODO, you should *expand* it to explain all of that. | |
| 19:39:10 | mriedem | making the --boot-from-volume optional more customizable like the --block-device-mapping option is not what we want, it defeats the purpose of a simple boot from volume option, | |
| 19:39:18 | efried | right | |
| 19:39:22 | mriedem | so that likely means adding bootindex= to --block-device-mapping | |
| 19:39:36 | sean-k-mooney | honestly the default shoudl proably be delete on instnace terminate | |
| 19:39:40 | mriedem | but that has implications for how the command currently requires you to specify --image|--image-property|--volume | |
| 19:39:45 | efried | you need a new option, --remove-volume-on-instance-deletion, which is only allowed with --boot-from-volume | |
| 19:40:01 | mriedem | i don't think that's much better ux | |
| 19:40:07 | efried | I'm not serious. | |
| 19:40:13 | mriedem | heh, ok | |
| 19:40:35 | efried | I know it's tough to tell with my rbf and deadpan humor style. | |
| 19:40:41 | artom | --boot-from-volume size=1,cleanup=plz | |
| 19:40:50 | mriedem | sean-k-mooney: if you have a dumb ass old app which stores data in the root volume, then deleting that by default is likely not what you want | |
| 19:40:57 | mriedem | artom: f and no | |
| 19:41:00 | artom | Conversly, --boot-from-volume size=1,cleanup=nowai | |
| 19:41:00 | mriedem | see what i just said above | |
| 19:41:04 | mriedem | no | |
| 19:41:05 | mriedem | no | |
| 19:41:11 | artom | Hahaha | |
| 19:41:13 | mriedem | that's what --block-device-mapping is for | |
| 19:41:14 | sean-k-mooney | mriedem: proably but we have had customer complain about this before | |
| 19:41:27 | sean-k-mooney | mainly because they expected it to delete by default | |
| 19:41:34 | mriedem | well they were wrong | |
| 19:41:37 | efried | OS_DELETE_VOLUME_WITH_INSTANCE=true openstack server create .... | |
| 19:41:37 | mriedem | bug report closed | |
| 19:41:50 | sean-k-mooney | yep and they acepted that | |
| 19:41:59 | efried | (the env var solution is just for dtroyer) | |
| 19:42:00 | mriedem | "please delete my data by default thanks!" | |
| 19:42:09 | artom | efried, we need to find a way to use /proc and or /sys for this | |
| 19:42:14 | efried | lol | |
| 19:42:26 | mriedem | or, | |
| 19:42:35 | mriedem | read in yaml support for the osc commands like kubectrl | |
| 19:42:35 | efried | I'm sure we could develop a hardware module | |
| 19:42:39 | mriedem | *kubectl | |
| 19:42:47 | artom | efried, oh oh oh! Cyborg with FPGAs | |
| 19:42:50 | mriedem | then you could define your block_device_mapping_v2 in yaml! | |
| 19:42:53 | efried | you mean a yaml front end to the REST API? | |
| 19:43:03 | mriedem | efried: just osc to start | |
| 19:43:11 | mriedem | i'm kidding | |
| 19:43:12 | mriedem | again | |
| 19:43:21 | sean-k-mooney | have we intoduced them to this thing call curl | |
| 19:43:39 | artom | Ya man, it's what I do in the squat rack every Friday | |
| 19:43:42 | sean-k-mooney | they can pass a json payload and be as declaritvie as they like | |
| 19:43:51 | eandersson | btw the Rescheduling Exceptions we saw were most likely due to missing records in placement. | |
| 19:44:08 | mriedem | hear me out, bring back xml support in the rest api | |
| 19:44:12 | mriedem | who's with me?! | |
| 19:44:13 | eandersson | We just didn't know until we deployed https://github.com/openstack/nova/commit/a5269012a3b442a9e4055a7d523faff45f105f2b | |
| 19:44:32 | sean-k-mooney | eandersson: missing compute node resouce providers? | |
| 19:45:01 | eandersson | Since we did a major jump (Mitaka -> Rocky) something went wrong when we first ran heal_allocations | |
| 19:45:05 | eandersson | So a lot of VMs were missing | |
| 19:45:24 | sean-k-mooney | oh ok | |
| 19:45:37 | sean-k-mooney | so it was missing the allocations not the RPs | |
| 19:45:41 | eandersson | Yea | |
| 19:45:51 | eandersson | We ran it, but not sure why we didn't notice that it failed. | |
| 19:45:56 | eandersson | --dry-run will help A LOT with this | |
| 19:47:02 | eandersson | We have probably 200 "bad" vms overall that is prevening heal_allocations from finishing, and currently fixing them one by one. | |
| 19:47:02 | sean-k-mooney | eandersson: while your here is https://blueprints.launchpad.net/nova/+spec/config-tsc-freq something you care about | |
| 19:47:39 | eandersson | Interesting | |
| 19:47:57 | sean-k-mooney | i need to go update that based on some irc converstation but we are suggesting adding 2 extraspcs/image properties | |
| 19:48:17 | mriedem | eandersson: don't forget to report a bug against heal_allocations for whatever you ran into re the ML thread from last night | |
| 19:48:34 | eandersson | Yea - started writing the bug reports | |
| 19:48:35 | sean-k-mooney | hw:inv_tsc=true|false and hw:tsc_freq_mhz=<frequency> | |
| 19:48:42 | mriedem | brb | |
| 19:48:47 | eandersson | but a lot of these issues are so specific to us I feel stupid writing the reports lol | |
| 19:49:08 | eandersson | like some of these "bad vms" are from like Juno | |
| 19:49:34 | sean-k-mooney | did any of the vms "disapear" after upgrading | |
| 19:49:35 | eandersson | or maybe even due to bad database surgery at some point | |
| 19:50:03 | eandersson | none disapeared | |
| 19:50:21 | eandersson | I think because we run NUMA a lot of issues are hidden from us. | |
| 19:50:26 | sean-k-mooney | ok there was a change around rocky where we start cleaning up falid evaucted instnace | |
| 19:50:38 | eandersson | Like the above would have been a major issue if it wasn't for NUMA, because when it fails due to the above it just reschedules. | |
| 19:51:07 | eandersson | We only notice it when it hits like 10+ failed reschedules and throws an exception | |
| 19:51:34 | sean-k-mooney | so numa made it better or worse? | |
| 19:51:34 | eandersson | and ironically we bumped it from 10 -> 20 failures because we didn't realize that placement was in an inconsistent state. | |
| 19:51:51 | eandersson | well it prevented computes from unintentially getting oversubscribed | |
| 19:52:13 | sean-k-mooney | i think that is a first. numa making the failure mode better :) | |
| 19:52:41 | sean-k-mooney | but ya the addtional tracking in the numa toplogy blob in the db would help with that | |
| 20:02:22 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove service_uuids_online_data_migration https://review.opendev.org/693421 | |
| 20:12:39 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove dead HostAPI.service_delete code https://review.opendev.org/693422 | |
| 20:35:22 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Use named kwargs in compute.API.resize https://review.opendev.org/693425 | |
| 20:36:14 | mriedem | god py38 non-voting talk already? | |
| 20:36:20 | mriedem | can we just skip to py500 already | |
| 20:45:12 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove now invalid TODO from ComputeManager._confirm_resize https://review.opendev.org/693427 | |