Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-09-19
16:39:33 sdague if they don't provide the url then the whole uwsgi thing can't work
16:42:12 openstackgerrit Eric Fried proposed openstack/nova master: Don't fix protocol-less glance api_servers anymore https://review.openstack.org/505317
16:42:17 efried sdague ^ Done.
16:43:13 sdague efried: ok, cool, last thing, I think we probably need a reno to tell people that they will hard fail if they only use IPs
16:43:40 efried sdague Okay.
16:49:56 openstackgerrit Eric Fried proposed openstack/nova master: Don't fix protocol-less glance api_servers anymore https://review.openstack.org/505317
16:49:57 efried sdague ^ Not real experienced with release notes, that okay?
16:51:39 sdague efried: wfm
16:52:13 dansmith jaypipes: you around?
16:58:50 openstackgerrit Eric Fried proposed openstack/nova-specs master: Spec: Use keystoneauth1 Adapter for endpoints https://review.openstack.org/500190
16:59:03 efried sdague ^ Added barbican. That sucker should be ready for review, I think.
16:59:17 efried mriedem FYI ^
17:31:45 cfriesen anyone know if region names are case sensitive?
17:32:13 openstackgerrit Ed Leafe proposed openstack/nova-specs master: Re-propose nested resource providers spec https://review.openstack.org/505209
17:32:45 edleafe cfriesen: I know that when I worked with RAX regions, they were
17:55:35 tasker the live-migration of the instance that was interrupted during the "post-migrtaion" tasks and the BINDING_PROFILE=None exception, the instance moved to the target host, but nova was interrupted and didn't update the database. nova thinks the instance is still on the original host. how can I get nova to update itself?
17:56:16 tasker restarting the `nova-compute` service did not do what I thought it might -- look to see what instances were active on the compute-host and update the tables accordingly.
17:58:09 tasker restarting results in "While synchronizing instance power states, found 2 instances in the database and 1 instances on the hypervisor." followed by "Instance is unexpectedly not found. Ignore."
17:58:43 tasker that's on the original host. the destination host has "While synchronizing instance power states, found 3 instances in the database and 4 instances on the hypervisor." and nothing else.
18:02:05 tasker i found https://bugs.launchpad.net/nova/+bug/1288958 on the same "Instance is unexpectedly not found. Igonre." message, but that one expired in April `16. the steps disclosed there are different but the state of Nova is similar.
18:02:10 openstack Launchpad bug 1288958 in OpenStack Compute (nova) "nova out of sync with the hypervisor " [Medium,Expired]
18:02:52 tasker nova list shows my particular instance with "power state = NOSTATE"
18:12:35 melwitt tasker: was it the 'setup_networks_on_host' that failed during post_live_migration_at_destination? because if so, that means the network failed setup on the destination and probably isn't working properly on the target
18:13:32 melwitt (I would think, I'm not a live migration expert)
18:20:11 tasker I don't see `setup_networks_on_host` in the traceback. it's "post_live_migration_at_destination -> migrate_instance_finish -> _update_port_binding_for_instance"
18:20:29 tasker do you have a pastie site that you prefer to use? I can flip you the traceback.
18:22:19 melwitt okay, I see the network_api.migrate_instance_finish call (just looking at the code). it's part of the network setup AFAICT
18:23:11 melwitt we use paste.openstack.org and pastebin.com
18:24:20 melwitt but just looking at the code, you're seeing it fail during the network setup at the destination, which would mean it's not going to work properly there, so it's not updating it to the target
18:25:57 melwitt yeah, the migrate_instance_finish is what updates the port binding for the instance (see nova/network/neutronv2/api.py)
18:27:46 melwitt if that failed, I don't think it would be correct to update the instance host to the target host. so AFAICT, the instance is still on the original host and there's a half-baked instance on the destination which needs to be cleaned up
18:28:16 tasker virsh list shows no instance on the original host.
18:28:49 tasker it does show it at the target host, but I cannot tell what state it's in.
18:28:52 melwitt *looking at more code* yeah, I was trying to see what has happened to the libvirt domain by this point
18:31:33 tasker melwitt: http://paste.openstack.org/show/621469/
18:31:38 melwitt so it moved the domain, failed to update the port binding, so it's not in a working state. so far I'm not seeing how you could recover from that
18:32:13 tasker deletion is recovery, right? <G>
18:33:45 melwitt heh
18:34:06 openstackgerrit Claudiu Belu proposed openstack/nova master: hyperv: report disk_available_least field https://review.openstack.org/504904
18:38:03 tasker this is a test cluster so no real worries.
18:42:00 melwitt okay, well that's good. the live migration flow is known to be gnarly and if anything fails in the middle of it, manual intervention is needed to fix things. I'm not familiar with more detail about it
18:42:35 melwitt from the trace you posted though, it looks like there's a bug in assuming binding_profile can't be None
18:43:18 melwitt er, actually it should be able to be assumed because: binding_profile = p.get(BINDING_PROFILE, {})
18:45:26 tasker yeah, mriedem and I have addressed that with https://review.openstack.org/#/c/504260
18:46:19 melwitt oh, I see. it's set but it's set to None. got it
18:46:46 tasker thanks for your help melwitt. I'm going to work on getting rid of this now broken instance.
18:53:36 openstackgerrit Merged openstack/nova stable/pike: Target context when setting instance to ERROR when over quota https://review.openstack.org/504178
19:03:15 sdague mriedem: https://review.openstack.org/#/c/454323/ that's the defaulting live snapshot to true
19:15:05 tasker I've got one instance left that's not migrating. the migration request is accepted by the API server, but after that it just disappears. `nova migration-list` immediately show is as error but I see no errors logged in the api containers and the message doesn't get to the compute hosts. I'm stumped on this one. I have no tracebacks or logs to show.
19:15:22 tasker messages disappearing are usually related to rabbit communications.
19:15:32 tasker but all other migrations worked just fine. except this instance.
19:16:14 tasker this instance was made when the cluster was at mitaka / trusty.
19:41:16 melwitt tasker: I think next it goes to conductor so check your nova-conductor logs for errors
19:41:34 tasker I have not. I will check now.
19:42:45 tasker you are correct. looking deeper.
19:43:06 tasker g'ah. "no valid host was found".
19:43:49 tasker same, regardless of which host I try and send it to. ( i only have 3 hypervisors ).
19:45:01 melwitt okay, to dig further, you need to look at nova-scheduler debug logs. if you don't already have log level DEBUG enabled, you'll need to configure that and restart nova-scheduler
19:45:14 melwitt there you would be able to see which scheduler filters failed and why
19:49:47 openstackgerrit Dan Smith proposed openstack/nova master: Add base implementation for efficient cross-cell instance listing https://review.openstack.org/504983
19:49:47 openstackgerrit Dan Smith proposed openstack/nova master: Make instance_list honor global query limit https://review.openstack.org/504984
19:49:48 openstackgerrit Dan Smith proposed openstack/nova master: Add db.instance_get_by_sort_filters() https://review.openstack.org/504985
19:49:48 openstackgerrit Dan Smith proposed openstack/nova master: Support pagination in instance_list https://review.openstack.org/504986
19:49:49 openstackgerrit Dan Smith proposed openstack/nova master: Add fault-filling into instance_get_all_by_filters_sort() https://review.openstack.org/505391
19:49:49 openstackgerrit Dan Smith proposed openstack/nova master: Add tests to validate instance_list handles faults correctly https://review.openstack.org/505392
19:57:11 sdague efried: looks like you have to fix some unit tests on the remove
19:57:29 efried sdague Sorry, which change set are we talking about?
19:57:47 efried oh, https://review.openstack.org/#/c/505317/ ?
19:57:56 sdague yeh
19:58:21 efried bah. I took a quick look for stuff I thought might be affected, but didn't run full tox locally.
19:58:28 efried You know, cause of the heat footprint :)
19:58:32 efried will fix.
19:58:34 sdague heh
20:15:45 dansmith gdi, pep8.. good for nothing.
21:01:19 openstackgerrit Eric Fried proposed openstack/nova master: Don't fix protocol-less glance api_servers anymore https://review.openstack.org/505317
21:01:23 mriedem is it just me or is the live migration job failing a lot now
21:01:40 efried sdague -^ fixed, I believe.
21:09:02 mikal mriedem: yeah, I should send that heads up email as discussed. I will do that tonight / tomorrow.
21:09:17 mikal In other news, is there a way to search for patches containing certain text?
21:09:28 mikal I want to find the couple of patches that renaming the privsep contexts broke
21:09:33 mriedem mikal: i did put something about privsep in my last recap email
21:09:45 mriedem mikal: search in the commit message?
21:10:05 mikal mriedem: yeah, I saw your para. I'll do something more explicit as well.
21:10:22 mikal mriedem: it might not be in the commit message though. I suspect I get to download every outstanding ref and grep them.
21:12:10 mriedem ask clarkb
21:12:50 clarkb mikal: mriedem https://review.openstack.org/Documentation/user-search.html
21:13:17 openstackgerrit Dan Smith proposed openstack/nova master: Add base implementation for efficient cross-cell instance listing https://review.openstack.org/504983
21:13:17 openstackgerrit Dan Smith proposed openstack/nova master: Make instance_list honor global query limit https://review.openstack.org/504984
21:13:18 openstackgerrit Dan Smith proposed openstack/nova master: Add db.instance_get_by_sort_filters() https://review.openstack.org/504985
21:13:18 openstackgerrit Dan Smith proposed openstack/nova master: Support pagination in instance_list https://review.openstack.org/504986
21:13:19 openstackgerrit Dan Smith proposed openstack/nova master: Add fault-filling into instance_get_all_by_filters_sort() https://review.openstack.org/505391
21:13:19 openstackgerrit Dan Smith proposed openstack/nova master: Add tests to validate instance_list handles faults correctly https://review.openstack.org/505392
21:13:56 mikal clarkb: I don't see a "contains text" or equivalent there. Am I going blind?
21:14:49 clarkb I'm not sure
21:14:57 clarkb we just upgraded and I haven't had a chance t oread the new search stuff
21:15:58 clarkb looks like you can do path, filename, commit message, and diff lines, but not diff line content
21:16:27 mikal clarkb: yeah, so that doesn't save me
21:16:39 clarkb codesearch.openstack.org may be of use
21:16:41 mikal clarkb: I might script downloading every patch as a diff and then just grep
21:16:51 mikal Does codesearch do unmerged patches?

Earlier   Later