| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-09-11 | |||
| 21:37:42 | mriedem | i just wanted to make sure we realize what we're signing up for | |
| 21:38:11 | dansmith | I guess I don't see it as a change from what we've been doing | |
| 21:38:33 | mriedem | we haven't been doing offline migrations | |
| 21:38:39 | melwitt | yeah, this one is the only one we haven't provided a manual run for, right? | |
| 21:38:56 | melwitt | that is, provided a nova-manage way to run it (via online_data_migrations) | |
| 21:39:26 | dansmith | we had one previously for pci stuff that was like the very first one | |
| 21:39:51 | mriedem | we have 3 other online data migrations added in pike, why wouldn't we also be providing offline options for those? | |
| 21:39:55 | dansmith | mriedem: we /have/ been providing them by having this command | |
| 21:40:11 | melwitt | yeah, that's what nova-manage online_data_migrations is | |
| 21:40:21 | dansmith | yeah | |
| 21:40:22 | dansmith | exactly | |
| 21:40:36 | melwitt | what we haven't had the past few times is actual active online data migrations (sorry that's confusing) | |
| 21:40:37 | dansmith | it's the offline way to run online migrations ... always have been | |
| 21:41:03 | dansmith | yeah, some of the ones we've had recently didn't actually do online background stuff and only happened as part of this command, but.. it's the same thing | |
| 21:41:10 | melwitt | right | |
| 21:41:24 | mriedem | huh? | |
| 21:41:38 | mriedem | so we've always required that everything is down when we run online_data_migrations? | |
| 21:41:51 | melwitt | like, the past few data migrations we did, were not done actively in the background automatically while nova runs | |
| 21:41:58 | mriedem | https://docs.openstack.org/nova/latest/user/upgrade.html#rolling-upgrade-process | |
| 21:42:08 | mriedem | "Start all services on the new code, with [upgrade_levels]compute=auto in nova.conf. It is safest to start nova-conductor first and nova-api last." | |
| 21:42:29 | melwitt | no, we don't require that things are down. it's just that we aren't doing it automatically in the background, we provided only the manual nova-manage command to take care of it | |
| 21:42:31 | mriedem | (later step): "This process can put significant extra write load on the database. Complete all online data migrations using: nova-manage db online_data_migrations --max-count <number>. " | |
| 21:42:47 | dansmith | I think we're stuck on terminology here | |
| 21:43:07 | dansmith | mriedem: where are you right now? we're about done here and this might be easier in person | |
| 21:43:26 | mriedem | i'm cranky now and hiding | |
| 21:43:29 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/502700 | |
| 21:43:32 | dansmith | alright | |
| 21:46:26 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: doc: note that custom resources are not fully supported https://review.openstack.org/501252 | |
| 21:46:38 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif master: Updated from global requirements https://review.openstack.org/502708 | |
| 21:47:24 | mriedem | i'm on a couch in a corner by the EMC business area | |
| 21:49:09 | openstackgerrit | OpenStack Proposal Bot proposed openstack/python-novaclient master: Updated from global requirements https://review.openstack.org/502745 | |
| 21:54:16 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/pike: doc: note that custom resources are not fully supported https://review.openstack.org/500521 | |
| 22:05:16 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Refactor ServerMovingTests for non-move tests https://review.openstack.org/498596 | |
| 22:38:28 | rybridges | Hello. I had a question or 2 about vendordata in the Ocata release | |
| 22:39:39 | rybridges | Are we still able to write our vendordata logic stuff into a driver? I am noticing this line in the ocata code base -> https://github.com/openstack/nova/blob/stable/ocata/nova/api/metadata/base.py#L745 Is vendordata not doable through a driver anymore? | |
| 23:07:20 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Merge extended availability zone controller https://review.openstack.org/502859 | |
| #openstack-nova - 2017-09-12 | |||
| 00:06:50 | mwynne | Hey guys. I'm looking at the rabbit queues (rabbitmqctl list_queues | awk '$2 != 0{print}'), and noticed that the conductor queue is always increasing. Is this normal? Can I debug this somehow? | |
| 00:12:37 | mwynne | netstat -tunlp | |
| 00:12:47 | mwynne | (whoops) | |
| 04:09:16 | openstackgerrit | Merged openstack/nova master: Refactor ServerMovingTests for non-move tests https://review.openstack.org/498596 | |
| 04:21:31 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/502700 | |
| 07:23:04 | rabel | can someone review https://review.openstack.org/#/c/494169/ ? i think except from minor changes it is ready for weeks. | |
| 08:02:11 | openstackgerrit | Christian Berendt proposed openstack/nova master: Choose VMware datastore by the least provisioned space https://review.openstack.org/402632 | |
| 08:51:25 | rabel | https://review.openstack.org/#/c/402632/10 seems to be ready for final review for months. could someone have a look at it? | |
| 10:07:45 | mewald | Hey guys, is there a chance the block_device_mapping table can be truncated / cleared without any problems? Nova should be able to get this info from Cinder via API, right? | |
| 10:28:38 | mewald | ok forget it, it was bullshit | |
| 10:34:01 | mewald | I have a wrong / incomplete connection_info column in block_device_mapping table. New volumes are added with different connection_info JSON objects than: for example they include the secret_uuid. The lack of this uuid is currently breaking my set up . Any advice on how to fix this? | |
| 11:58:45 | bhagyashris | Hi all, Need guidance of How to write the upgrade script? | |
| 13:29:18 | boolman | I'm having problems adding iolimits on my vms. I tried adding quota on flavor,image,volume but it won't add iotune to the virsh DOM. output from image/volume/flavor show : http://ix.io/zMX | |
| 13:52:53 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/nova-specs master: Intel Fortville Dynamic Device Personalization (DDP) https://review.openstack.org/503001 | |
| 14:06:15 | openstackgerrit | Merged openstack/python-novaclient master: Updated from global requirements https://review.openstack.org/502745 | |
| 14:07:57 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/502700 | |
| 14:09:05 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/nova-specs master: Intel Fortville Dynamic Device Personalization (DDP) https://review.openstack.org/503001 | |
| 14:11:07 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/os-vif master: Migration from ``ip`` commands to ``pyroute2`` https://review.openstack.org/484386 | |
| 14:18:58 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Transform instance.resize.error notifications https://review.openstack.org/417882 | |
| 14:20:05 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: fake_notifier: Refactor wait_for_versioned_notification https://review.openstack.org/489637 | |
| 14:38:44 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/nova-specs master: Intel Fortville Dynamic Device Personalization (DDP) https://review.openstack.org/503001 | |
| 15:00:53 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Enable cold migration with target host(1/2) https://review.openstack.org/408955 | |
| 15:06:12 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Enable cold migration with target host(2/2) https://review.openstack.org/408964 | |
| 15:06:29 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: api-ref: Add parameters in cold migrate action https://review.openstack.org/410042 | |
| 15:14:14 | mriedem | bauzas: are you ok with this? https://review.openstack.org/#/c/499399/ - you said you were happy after the reno got updated | |
| 15:14:24 | mriedem | i'd like to move this series along to get it backported to stable/pike | |
| 15:17:51 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove dest node allocation if evacuate MoveClaim fails https://review.openstack.org/499878 | |
| 15:17:52 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Pass migration from API to conductor for evacuate https://review.openstack.org/500176 | |
| 15:17:52 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add a test to make sure failed evacuate cleans up dest allocation https://review.openstack.org/499877 | |
| 15:17:53 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add recreate test for evacuate claim failure https://review.openstack.org/499874 | |
| 15:17:53 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Create allocations against forced dest host during evacuate https://review.openstack.org/499399 | |
| 15:17:54 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Modernize set_vm_state_and_notify https://review.openstack.org/499799 | |
| 15:31:19 | mriedem | Kevin_Zheng: yes if you go overquota after the build_request is created when we recheck in conductor, the instance will be put into ERROR state https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1084 | |
| 15:31:20 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Moving more utils to ServerResourceAllocationTestBase https://review.openstack.org/499539 | |
| 15:31:20 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: factor out compute service start in ServerMovingTest https://review.openstack.org/503037 | |
| 15:31:36 | mriedem | Kevin_Zheng: that's because we can't just delete the instance at that point in conductor | |
| 15:31:53 | mriedem | because the use could be listing the instance via the BuildRequest before it goes to ERROR state | |
| 15:31:56 | mriedem | *user | |
| 15:31:59 | Kevin_Zheng | mriedem: I understand, but isn't that a big change to users? | |
| 15:32:49 | Kevin_Zheng | If large number of requests comes at the same time, in the past, I got OverQuota error | |
| 15:32:57 | mriedem | well, depends on what happens before counting quotas if you passed the initial check in the api and then failed when committing the reservation down in the compute | |
| 15:33:03 | Kevin_Zheng | But now, I got huge number of instances | |
| 15:33:12 | Kevin_Zheng | in Error States | |
| 15:33:30 | mriedem | Kevin_Zheng: are you requesting multiple instances in the same request or concurrent requests? | |
| 15:33:39 | Kevin_Zheng | concurrent requests | |
| 15:34:51 | mriedem | so in the past we'd create a reservation in the api and the quota check would include reservations | |
| 15:35:04 | mriedem | now we don't have reservations, and we're counting based on the instances in the cells, | |
| 15:35:23 | mriedem | i think what might be missing, and this might have come up in reviewing the series, was we don't count the build requests in the api db | |
| 15:35:26 | mriedem | but i need to look | |
| 15:35:31 | mriedem | unless melwitt is around | |
| 15:35:58 | Kevin_Zheng | Yeah, that's also what I thought as a solution :) | |
| 15:37:10 | mriedem | melwitt: dansmith: do you remember talking about including a count of build_requests in the API during the quota check? | |
| 15:37:43 | dansmith | mriedem: I think I remember talking about it | |
| 15:39:03 | mriedem | right so if i create 20 instances in concurrent requests, i'll probably get 10 in ERROR state | |
| 15:39:10 | mriedem | rather than 10 plus 10 failed requests with 409s | |
| 15:40:12 | mriedem | Kevin_Zheng: we should start with a bug report, can you open one? | |
| 15:40:27 | Kevin_Zheng | mriedem: sure, I will | |
| 15:40:51 | mriedem | check_num_instances_quota is just iterating the cells looking for instances, but at this point we don't have instances in the cells, we have build_requests in the api db | |
| 15:41:01 | mriedem | so we'd need something similar | |
| 15:42:44 | mriedem | and that method is called from the api and conductor, and in the case of conductor we'd have to not count build_requests because at that point that conductor calls check_num_instances_quota we have the instances created in the cells | |
| 15:43:11 | openstackgerrit | Ghanshyam Mann proposed openstack/nova-specs master: Spec to remove the hide server address config options Partial implement blueprint remove-configurable-hide-server-address-feature https://review.openstack.org/502516 | |
| 15:47:21 | mriedem | i see one issue with counting via build_requests is that when we count instances in the api, we also count cores and ram, which are fields on the instance in the db, | |
| 15:47:44 | mriedem | with a build_request, we have a serialized instance record, so we can count total instances via build_requests, but not cores/ram...unless we used placement | |
| 15:48:09 | mriedem | we want to use placement for counting quotas anyway...which we are going to talk about tomorrow morning | |