| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-07 | |||
| 19:40:35 | efried | I know it's tough to tell with my rbf and deadpan humor style. | |
| 19:40:41 | artom | --boot-from-volume size=1,cleanup=plz | |
| 19:40:50 | mriedem | sean-k-mooney: if you have a dumb ass old app which stores data in the root volume, then deleting that by default is likely not what you want | |
| 19:40:57 | mriedem | artom: f and no | |
| 19:41:00 | artom | Conversly, --boot-from-volume size=1,cleanup=nowai | |
| 19:41:00 | mriedem | see what i just said above | |
| 19:41:04 | mriedem | no | |
| 19:41:05 | mriedem | no | |
| 19:41:11 | artom | Hahaha | |
| 19:41:13 | mriedem | that's what --block-device-mapping is for | |
| 19:41:14 | sean-k-mooney | mriedem: proably but we have had customer complain about this before | |
| 19:41:27 | sean-k-mooney | mainly because they expected it to delete by default | |
| 19:41:34 | mriedem | well they were wrong | |
| 19:41:37 | efried | OS_DELETE_VOLUME_WITH_INSTANCE=true openstack server create .... | |
| 19:41:37 | mriedem | bug report closed | |
| 19:41:50 | sean-k-mooney | yep and they acepted that | |
| 19:41:59 | efried | (the env var solution is just for dtroyer) | |
| 19:42:00 | mriedem | "please delete my data by default thanks!" | |
| 19:42:09 | artom | efried, we need to find a way to use /proc and or /sys for this | |
| 19:42:14 | efried | lol | |
| 19:42:26 | mriedem | or, | |
| 19:42:35 | mriedem | read in yaml support for the osc commands like kubectrl | |
| 19:42:35 | efried | I'm sure we could develop a hardware module | |
| 19:42:39 | mriedem | *kubectl | |
| 19:42:47 | artom | efried, oh oh oh! Cyborg with FPGAs | |
| 19:42:50 | mriedem | then you could define your block_device_mapping_v2 in yaml! | |
| 19:42:53 | efried | you mean a yaml front end to the REST API? | |
| 19:43:03 | mriedem | efried: just osc to start | |
| 19:43:11 | mriedem | i'm kidding | |
| 19:43:12 | mriedem | again | |
| 19:43:21 | sean-k-mooney | have we intoduced them to this thing call curl | |
| 19:43:39 | artom | Ya man, it's what I do in the squat rack every Friday | |
| 19:43:42 | sean-k-mooney | they can pass a json payload and be as declaritvie as they like | |
| 19:43:51 | eandersson | btw the Rescheduling Exceptions we saw were most likely due to missing records in placement. | |
| 19:44:08 | mriedem | hear me out, bring back xml support in the rest api | |
| 19:44:12 | mriedem | who's with me?! | |
| 19:44:13 | eandersson | We just didn't know until we deployed https://github.com/openstack/nova/commit/a5269012a3b442a9e4055a7d523faff45f105f2b | |
| 19:44:32 | sean-k-mooney | eandersson: missing compute node resouce providers? | |
| 19:45:01 | eandersson | Since we did a major jump (Mitaka -> Rocky) something went wrong when we first ran heal_allocations | |
| 19:45:05 | eandersson | So a lot of VMs were missing | |
| 19:45:24 | sean-k-mooney | oh ok | |
| 19:45:37 | sean-k-mooney | so it was missing the allocations not the RPs | |
| 19:45:41 | eandersson | Yea | |
| 19:45:51 | eandersson | We ran it, but not sure why we didn't notice that it failed. | |
| 19:45:56 | eandersson | --dry-run will help A LOT with this | |
| 19:47:02 | eandersson | We have probably 200 "bad" vms overall that is prevening heal_allocations from finishing, and currently fixing them one by one. | |
| 19:47:02 | sean-k-mooney | eandersson: while your here is https://blueprints.launchpad.net/nova/+spec/config-tsc-freq something you care about | |
| 19:47:39 | eandersson | Interesting | |
| 19:47:57 | sean-k-mooney | i need to go update that based on some irc converstation but we are suggesting adding 2 extraspcs/image properties | |
| 19:48:17 | mriedem | eandersson: don't forget to report a bug against heal_allocations for whatever you ran into re the ML thread from last night | |
| 19:48:34 | eandersson | Yea - started writing the bug reports | |
| 19:48:35 | sean-k-mooney | hw:inv_tsc=true|false and hw:tsc_freq_mhz=<frequency> | |
| 19:48:42 | mriedem | brb | |
| 19:48:47 | eandersson | but a lot of these issues are so specific to us I feel stupid writing the reports lol | |
| 19:49:08 | eandersson | like some of these "bad vms" are from like Juno | |
| 19:49:34 | sean-k-mooney | did any of the vms "disapear" after upgrading | |
| 19:49:35 | eandersson | or maybe even due to bad database surgery at some point | |
| 19:50:03 | eandersson | none disapeared | |
| 19:50:21 | eandersson | I think because we run NUMA a lot of issues are hidden from us. | |
| 19:50:26 | sean-k-mooney | ok there was a change around rocky where we start cleaning up falid evaucted instnace | |
| 19:50:38 | eandersson | Like the above would have been a major issue if it wasn't for NUMA, because when it fails due to the above it just reschedules. | |
| 19:51:07 | eandersson | We only notice it when it hits like 10+ failed reschedules and throws an exception | |
| 19:51:34 | sean-k-mooney | so numa made it better or worse? | |
| 19:51:34 | eandersson | and ironically we bumped it from 10 -> 20 failures because we didn't realize that placement was in an inconsistent state. | |
| 19:51:51 | eandersson | well it prevented computes from unintentially getting oversubscribed | |
| 19:52:13 | sean-k-mooney | i think that is a first. numa making the failure mode better :) | |
| 19:52:41 | sean-k-mooney | but ya the addtional tracking in the numa toplogy blob in the db would help with that | |
| 20:02:22 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove service_uuids_online_data_migration https://review.opendev.org/693421 | |
| 20:12:39 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove dead HostAPI.service_delete code https://review.opendev.org/693422 | |
| 20:35:22 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Use named kwargs in compute.API.resize https://review.opendev.org/693425 | |
| 20:36:14 | mriedem | god py38 non-voting talk already? | |
| 20:36:20 | mriedem | can we just skip to py500 already | |
| 20:45:12 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove now invalid TODO from ComputeManager._confirm_resize https://review.opendev.org/693427 | |
| 20:54:16 | openstackgerrit | Merged openstack/nova master: Execute CrossCellMigrationTask from MigrationTask https://review.opendev.org/635668 | |
| 20:54:23 | openstackgerrit | Merged openstack/nova master: Refresh instance in MigrationTask.execute Exception handler https://review.opendev.org/669012 | |
| 21:24:14 | mriedem | man test_detach_volume_shelved_or_offload_server seems to fail a lot | |
| 21:26:45 | openstackgerrit | François Palin proposed openstack/nova master: Add retry to cinder api calls related to volume detach https://review.opendev.org/669674 | |
| 21:28:26 | eandersson | Interesting - a VM without an allocation that is migrated also does not get an allocation | |
| 21:28:38 | eandersson | Would expected it to get "repaired" :p | |
| 21:28:41 | openstack | bug 1824435 in OpenStack Compute (nova) stein "fill_virtual_interface_list migration fails on second attempt" [Medium,In progress] https://launchpad.net/bugs/1824435 - Assigned to melanie witt (melwitt) | |
| 21:28:41 | openstackgerrit | Merged openstack/nova stable/stein: Add regression test for bug 1824435 https://review.opendev.org/692906 | |
| 21:28:44 | openstackgerrit | Merged openstack/nova stable/stein: Remove redundant call to get/create default security group https://review.opendev.org/692907 | |
| 21:31:04 | eandersson | melwitt, is that a bug in rocky as well? | |
| 21:31:20 | melwitt | no, stein is the earliest it shows up | |
| 21:36:13 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Implement update_provider_tree for mocked driver in test_resource_tracker https://review.opendev.org/693431 | |
| 21:37:42 | mriedem | eandersson: you should see this in the conductor logs https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/tasks/migrate.py#L51 | |
| 21:38:17 | mriedem | i would expect the scheduler to create allocations for the instance against the dest node | |
| 21:38:29 | mriedem | and those to stick if the migration completes successfully | |
| 21:38:48 | eandersson | We basically moved a site from one DC to another | |
| 21:38:58 | eandersson | and migrated all VMs | |
| 21:39:18 | ccstone | we didn't have nova in debug so we wouldn't have seen that message :/ | |
| 21:39:21 | eandersson | all of them was "healed" using the heal_allocations | |
| 21:40:29 | mriedem | ok, i don't know what would be going on there, but i'd expect the migration to heal the instance allocations as well | |
| 21:40:37 | mriedem | s/heal/recreate on dest node/ | |
| 21:41:44 | eandersson | Of course we are basing this on the heal_allocations output | |
| 21:45:14 | openstackgerrit | François Palin proposed openstack/nova master: Add retry to cinder api calls related to volume detach https://review.opendev.org/669674 | |
| 21:46:17 | mriedem | so the vm didn't have allocations, you migrated it successfully (and confirmed the migration), and ran heal allocations and it healed it rather than saying there was nothing to do? | |
| 21:47:03 | eandersson | pretty much yes | |
| 21:47:14 | mriedem | fake news! | |
| 21:47:16 | eandersson | haha | |